2023-11-17T06:35:54.940 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-17T06:35:54.940 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312 --description rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} --verbose --name yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/orig.config.yaml 2023-11-17T06:35:54.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T06:35:55.064 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312 branch: wip-yuri3-testing-2023-11-14-1227-pacific description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} email: null first_in_suite: false job_id: '7461312' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri3-testing-2023-11-14-1227-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac kernel: hwe: true workunit: branch: wip-yuri3-testing-2023-11-14-1227-pacific sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 7739 sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-11-14-1227-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip: null - cephadm.shell: host.a: - ceph orch device ls --refresh - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.apply: specs: - placement: count: 2 service_id: foo service_type: nfs spec: port: 12049 - service_id: nfs.foo service_type: ingress spec: backend_service: nfs.foo frontend_port: 2049 monitor_port: 9002 virtual_ip: '{{VIP0}}/{{VIPPREFIXLEN}}' - cephadm.wait_for_service: service: nfs.foo - cephadm.wait_for_service: service: ingress.nfs.foo - cephadm.shell: host.a: - ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - vip.exec: host.a: - mkdir /mnt/foo - sleep 5 - mount -t nfs {{VIP0}}:/fake /mnt/foo - echo test > /mnt/foo/testfile - sync - cephadm.shell: host.a: - "echo \"Check with each haproxy down in turn...\"\nfor haproxy in `ceph orch\ \ ps | grep ^haproxy.nfs.foo. | awk '{print $1}'`; do\n ceph orch daemon stop\ \ $haproxy\n while ! ceph orch ps | grep $haproxy | grep stopped; do sleep\ \ 1 ; done\n cat /mnt/foo/testfile\n echo $haproxy > /mnt/foo/testfile\n \ \ sync\n ceph orch daemon start $haproxy\n while ! ceph orch ps | grep $haproxy\ \ | grep running; do sleep 1 ; done\ndone\n" volumes: - /mnt/foo:/mnt/foo - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_22:29:26 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-17T06:35:55.064 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa; will attempt to use it 2023-11-17T06:35:55.065 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks 2023-11-17T06:35:55.065 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-17T06:35:55.067 INFO:teuthology.task.internal:Checking packages... 2023-11-17T06:35:55.089 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac' 2023-11-17T06:35:55.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-17T06:35:55.089 INFO:teuthology.packaging:ref: None 2023-11-17T06:35:55.089 INFO:teuthology.packaging:tag: None 2023-11-17T06:35:55.089 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T06:35:55.089 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T06:35:55.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T06:35:55.351 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-171-ged3c8e92-1focal 2023-11-17T06:35:55.351 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-17T06:35:55.357 INFO:teuthology.task.internal:no buildpackages task found 2023-11-17T06:35:55.357 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-17T06:35:55.376 INFO:teuthology.task.internal:Saving configuration 2023-11-17T06:35:55.390 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-17T06:35:55.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T06:35:55.419 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 06:29:53.640601', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-17T06:35:55.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 06:29:53.639356', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-11-17T06:35:55.445 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-17T06:35:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['host.a', 'client.0'] 2023-11-17T06:35:55.451 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['host.b', 'client.1'] 2023-11-17T06:35:55.451 INFO:teuthology.run_tasks:Running task console_log... 2023-11-17T06:35:55.518 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3647e29040>, signals=[15]) 2023-11-17T06:35:55.518 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-17T06:35:55.524 INFO:teuthology.task.internal:Opening connections... 2023-11-17T06:35:55.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T06:35:55.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:35:55.577 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2023-11-17T06:35:55.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:35:55.626 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-17T06:35:55.632 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-11-17T06:35:55.653 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-11-17T06:35:55.653 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-11-17T06:35:55.700 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Ubuntu" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:ID=ubuntu 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE=debian 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="20.04" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_CODENAME=focal 2023-11-17T06:35:55.701 INFO:teuthology.orchestra.run.smithi097.stdout:UBUNTU_CODENAME=focal 2023-11-17T06:35:55.702 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-11-17T06:35:55.724 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2023-11-17T06:35:55.731 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2023-11-17T06:35:55.731 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2023-11-17T06:35:55.779 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Ubuntu" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:ID=ubuntu 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE=debian 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="20.04" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_CODENAME=focal 2023-11-17T06:35:55.780 INFO:teuthology.orchestra.run.smithi193.stdout:UBUNTU_CODENAME=focal 2023-11-17T06:35:55.781 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-11-17T06:35:55.807 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-17T06:35:55.819 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-17T06:35:55.825 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-17T06:35:55.825 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2023-11-17T06:35:55.827 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2023-11-17T06:35:55.833 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-17T06:35:55.840 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-17T06:35:55.840 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2023-11-17T06:35:55.876 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2023-11-17T06:35:55.889 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-17T06:35:55.938 INFO:teuthology.run_tasks:Running task kernel... 2023-11-17T06:35:55.953 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-17T06:35:55.953 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2023-11-17T06:35:55.953 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.0': {'hwe': True}, 'host.b': {'hwe': True}, 'client.1': {'hwe': True}} 2023-11-17T06:35:55.953 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2023-11-17T06:35:55.953 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T06:35:55.954 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T06:35:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:35:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T06:35:55.967 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-11-17T06:35:55.967 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2023-11-17T06:35:55.973 INFO:teuthology.orchestra.run.smithi193.stdout:5.4.0-124-generic 2023-11-17T06:35:55.973 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.4.0-124-generic 2023-11-17T06:35:55.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-get clean 2023-11-17T06:35:56.018 INFO:teuthology.orchestra.run.smithi097.stdout:5.4.0-124-generic 2023-11-17T06:35:56.018 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.4.0-124-generic 2023-11-17T06:35:56.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get clean 2023-11-17T06:35:56.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-get update 2023-11-17T06:35:56.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2023-11-17T06:35:56.296 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-17T06:35:56.312 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-17T06:35:56.315 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-17T06:35:56.332 INFO:teuthology.orchestra.run.smithi193.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-11-17T06:35:56.368 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-17T06:35:56.386 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-17T06:35:56.386 INFO:teuthology.orchestra.run.smithi193.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-11-17T06:35:56.447 INFO:teuthology.orchestra.run.smithi193.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-11-17T06:35:56.679 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [907 kB] 2023-11-17T06:35:56.710 INFO:teuthology.orchestra.run.smithi193.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [907 kB] 2023-11-17T06:35:56.759 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,960 kB] 2023-11-17T06:35:56.796 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [482 kB] 2023-11-17T06:35:56.799 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-17T06:35:56.800 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-11-17T06:35:56.800 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,479 kB] 2023-11-17T06:35:56.820 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-17T06:35:56.822 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-11-17T06:35:56.822 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [755 kB] 2023-11-17T06:35:56.830 INFO:teuthology.orchestra.run.smithi097.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,131 kB] 2023-11-17T06:35:56.839 INFO:teuthology.orchestra.run.smithi097.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [270 kB] 2023-11-17T06:35:56.841 INFO:teuthology.orchestra.run.smithi097.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-17T06:35:56.842 INFO:teuthology.orchestra.run.smithi097.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-11-17T06:35:56.842 INFO:teuthology.orchestra.run.smithi097.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-11-17T06:35:56.842 INFO:teuthology.orchestra.run.smithi097.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-17T06:35:56.847 INFO:teuthology.orchestra.run.smithi097.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-17T06:35:56.847 INFO:teuthology.orchestra.run.smithi097.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-17T06:35:56.847 INFO:teuthology.orchestra.run.smithi097.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-17T06:35:56.864 INFO:teuthology.orchestra.run.smithi193.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,960 kB] 2023-11-17T06:35:56.864 INFO:teuthology.orchestra.run.smithi097.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-17T06:35:56.865 INFO:teuthology.orchestra.run.smithi097.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-17T06:35:56.865 INFO:teuthology.orchestra.run.smithi097.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-17T06:35:56.865 INFO:teuthology.orchestra.run.smithi097.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-17T06:35:56.951 INFO:teuthology.orchestra.run.smithi193.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [482 kB] 2023-11-17T06:35:56.960 INFO:teuthology.orchestra.run.smithi193.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-11-17T06:35:56.960 INFO:teuthology.orchestra.run.smithi193.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,479 kB] 2023-11-17T06:35:56.988 INFO:teuthology.orchestra.run.smithi097.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,570 kB] 2023-11-17T06:35:57.007 INFO:teuthology.orchestra.run.smithi193.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-11-17T06:35:57.007 INFO:teuthology.orchestra.run.smithi193.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [346 kB] 2023-11-17T06:35:57.011 INFO:teuthology.orchestra.run.smithi193.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-11-17T06:35:57.011 INFO:teuthology.orchestra.run.smithi193.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [755 kB] 2023-11-17T06:35:57.029 INFO:teuthology.orchestra.run.smithi193.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,131 kB] 2023-11-17T06:35:57.033 INFO:teuthology.orchestra.run.smithi193.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,570 kB] 2023-11-17T06:35:57.048 INFO:teuthology.orchestra.run.smithi193.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [270 kB] 2023-11-17T06:35:57.055 INFO:teuthology.orchestra.run.smithi193.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-11-17T06:35:57.059 INFO:teuthology.orchestra.run.smithi193.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-11-17T06:35:57.059 INFO:teuthology.orchestra.run.smithi193.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-11-17T06:35:57.059 INFO:teuthology.orchestra.run.smithi193.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-11-17T06:35:57.091 INFO:teuthology.orchestra.run.smithi193.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-11-17T06:35:57.091 INFO:teuthology.orchestra.run.smithi193.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-11-17T06:35:57.092 INFO:teuthology.orchestra.run.smithi193.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-11-17T06:35:57.093 INFO:teuthology.orchestra.run.smithi193.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-11-17T06:35:57.093 INFO:teuthology.orchestra.run.smithi193.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-11-17T06:35:57.095 INFO:teuthology.orchestra.run.smithi193.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-11-17T06:35:57.095 INFO:teuthology.orchestra.run.smithi193.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-11-17T06:35:57.471 INFO:teuthology.orchestra.run.smithi097.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [674 kB] 2023-11-17T06:35:57.486 INFO:teuthology.orchestra.run.smithi097.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-17T06:35:57.495 INFO:teuthology.orchestra.run.smithi097.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-17T06:35:57.495 INFO:teuthology.orchestra.run.smithi097.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-11-17T06:35:57.496 INFO:teuthology.orchestra.run.smithi097.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,367 kB] 2023-11-17T06:35:57.513 INFO:teuthology.orchestra.run.smithi193.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [674 kB] 2023-11-17T06:35:57.528 INFO:teuthology.orchestra.run.smithi193.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [398 kB] 2023-11-17T06:35:57.537 INFO:teuthology.orchestra.run.smithi193.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-11-17T06:35:57.537 INFO:teuthology.orchestra.run.smithi193.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,367 kB] 2023-11-17T06:35:57.598 INFO:teuthology.orchestra.run.smithi097.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-17T06:35:57.606 INFO:teuthology.orchestra.run.smithi097.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-11-17T06:35:57.607 INFO:teuthology.orchestra.run.smithi097.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [621 kB] 2023-11-17T06:35:57.612 INFO:teuthology.orchestra.run.smithi193.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-11-17T06:35:57.613 INFO:teuthology.orchestra.run.smithi193.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [329 kB] 2023-11-17T06:35:57.621 INFO:teuthology.orchestra.run.smithi193.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-11-17T06:35:57.621 INFO:teuthology.orchestra.run.smithi193.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [621 kB] 2023-11-17T06:35:57.634 INFO:teuthology.orchestra.run.smithi193.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [898 kB] 2023-11-17T06:35:57.635 INFO:teuthology.orchestra.run.smithi097.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [898 kB] 2023-11-17T06:35:57.817 INFO:teuthology.orchestra.run.smithi193.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-17T06:35:57.821 INFO:teuthology.orchestra.run.smithi097.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [188 kB] 2023-11-17T06:35:58.236 INFO:teuthology.orchestra.run.smithi193.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-17T06:35:58.238 INFO:teuthology.orchestra.run.smithi097.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-11-17T06:35:58.242 INFO:teuthology.orchestra.run.smithi193.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-11-17T06:35:58.244 INFO:teuthology.orchestra.run.smithi193.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-17T06:35:58.245 INFO:teuthology.orchestra.run.smithi097.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-11-17T06:35:58.252 INFO:teuthology.orchestra.run.smithi097.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-11-17T06:35:58.252 INFO:teuthology.orchestra.run.smithi193.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-17T06:35:58.254 INFO:teuthology.orchestra.run.smithi193.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-17T06:35:58.254 INFO:teuthology.orchestra.run.smithi097.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-11-17T06:35:58.256 INFO:teuthology.orchestra.run.smithi097.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-11-17T06:36:00.584 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 18.1 MB in 3s (6,996 kB/s) 2023-11-17T06:36:00.861 INFO:teuthology.orchestra.run.smithi193.stdout:Fetched 18.1 MB in 3s (6,667 kB/s) 2023-11-17T06:36:02.027 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-11-17T06:36:02.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2023-11-17T06:36:02.127 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-11-17T06:36:02.292 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-17T06:36:02.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2023-11-17T06:36:02.332 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2023-11-17T06:36:02.332 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2023-11-17T06:36:02.391 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-17T06:36:02.506 INFO:teuthology.orchestra.run.smithi097.stdout:The following additional packages will be installed: 2023-11-17T06:36:02.507 INFO:teuthology.orchestra.run.smithi097.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2023-11-17T06:36:02.507 INFO:teuthology.orchestra.run.smithi097.stdout: libusbmuxd6 linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-17T06:36:02.507 INFO:teuthology.orchestra.run.smithi097.stdout: linux-modules-extra-5.15.0-88-generic thermald upower usbmuxd 2023-11-17T06:36:02.508 INFO:teuthology.orchestra.run.smithi097.stdout:Suggested packages: 2023-11-17T06:36:02.508 INFO:teuthology.orchestra.run.smithi097.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2023-11-17T06:36:02.508 INFO:teuthology.orchestra.run.smithi097.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-88-generic 2023-11-17T06:36:02.562 INFO:teuthology.orchestra.run.smithi097.stdout:The following NEW packages will be installed: 2023-11-17T06:36:02.565 INFO:teuthology.orchestra.run.smithi097.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2023-11-17T06:36:02.565 INFO:teuthology.orchestra.run.smithi097.stdout: libusbmuxd6 linux-image-5.15.0-88-generic linux-image-generic-hwe-20.04 2023-11-17T06:36:02.566 INFO:teuthology.orchestra.run.smithi097.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-17T06:36:02.566 INFO:teuthology.orchestra.run.smithi097.stdout: thermald upower usbmuxd 2023-11-17T06:36:02.594 INFO:teuthology.orchestra.run.smithi193.stdout:Building dependency tree... 2023-11-17T06:36:02.595 INFO:teuthology.orchestra.run.smithi193.stdout:Reading state information... 2023-11-17T06:36:02.749 INFO:teuthology.orchestra.run.smithi097.stdout:0 upgraded, 13 newly installed, 0 to remove and 281 not upgraded. 2023-11-17T06:36:02.749 INFO:teuthology.orchestra.run.smithi097.stdout:Need to get 90.8 MB of archives. 2023-11-17T06:36:02.749 INFO:teuthology.orchestra.run.smithi097.stdout:After this operation, 497 MB of additional disk space will be used. 2023-11-17T06:36:02.749 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2023-11-17T06:36:02.779 INFO:teuthology.orchestra.run.smithi193.stdout:The following additional packages will be installed: 2023-11-17T06:36:02.779 INFO:teuthology.orchestra.run.smithi193.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2023-11-17T06:36:02.780 INFO:teuthology.orchestra.run.smithi193.stdout: libusbmuxd6 linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-17T06:36:02.780 INFO:teuthology.orchestra.run.smithi193.stdout: linux-modules-extra-5.15.0-88-generic thermald upower usbmuxd 2023-11-17T06:36:02.781 INFO:teuthology.orchestra.run.smithi193.stdout:Suggested packages: 2023-11-17T06:36:02.781 INFO:teuthology.orchestra.run.smithi193.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2023-11-17T06:36:02.781 INFO:teuthology.orchestra.run.smithi193.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-88-generic 2023-11-17T06:36:02.835 INFO:teuthology.orchestra.run.smithi193.stdout:The following NEW packages will be installed: 2023-11-17T06:36:11.464 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [20.9 MB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [11.4 MB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [58.0 MB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.88.98~20.04.46 [2,632 B] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2023-11-17T06:36:11.488 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 90.8 MB in 4s (20.4 MB/s) 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libplist3:amd64. 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2023-11-17T06:36:11.489 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libupower-glib3:amd64. 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2023-11-17T06:36:11.490 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout: libusbmuxd6 linux-image-5.15.0-88-generic linux-image-generic-hwe-20.04 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout: thermald upower usbmuxd 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout:0 upgraded, 13 newly installed, 0 to remove and 281 not upgraded. 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout:Need to get 90.8 MB of archives. 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout:After this operation, 497 MB of additional disk space will be used. 2023-11-17T06:36:11.491 INFO:teuthology.orchestra.run.smithi193.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [20.9 MB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [11.4 MB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98~20.04.1 [58.0 MB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.88.98~20.04.46 [2,632 B] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Fetched 90.8 MB in 1s (71.2 MB/s) 2023-11-17T06:36:11.492 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libplist3:amd64. 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libupower-glib3:amd64. 2023-11-17T06:36:11.493 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2023-11-17T06:36:11.494 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2023-11-17T06:36:11.494 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-17T06:36:11.494 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../05-linux-modules-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:11.494 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:11.564 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-17T06:36:11.576 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../06-linux-image-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:11.688 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:11.694 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-17T06:36:11.710 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../05-linux-modules-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:11.752 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:12.200 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-17T06:36:12.213 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:12.250 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:14.228 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-17T06:36:14.245 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../06-linux-image-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:14.348 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:14.856 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-17T06:36:14.869 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98~20.04.1_amd64.deb ... 2023-11-17T06:36:14.906 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:19.126 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2023-11-17T06:36:19.141 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.88.98~20.04.46_amd64.deb ... 2023-11-17T06:36:19.184 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.88.98~20.04.46) ... 2023-11-17T06:36:19.545 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package libevdev2:amd64. 2023-11-17T06:36:19.562 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2023-11-17T06:36:19.603 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2023-11-17T06:36:19.965 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package thermald. 2023-11-17T06:36:19.978 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2023-11-17T06:36:20.015 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2023-11-17T06:36:20.528 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package upower. 2023-11-17T06:36:20.542 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2023-11-17T06:36:20.589 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking upower (0.99.11-1build2) ... 2023-11-17T06:36:20.974 INFO:teuthology.orchestra.run.smithi193.stdout:Selecting previously unselected package usbmuxd. 2023-11-17T06:36:20.990 INFO:teuthology.orchestra.run.smithi193.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2023-11-17T06:36:21.032 INFO:teuthology.orchestra.run.smithi193.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2023-11-17T06:36:21.337 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2023-11-17T06:36:21.463 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2023-11-17T06:36:21.590 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2023-11-17T06:36:21.716 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2023-11-17T06:36:21.842 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2023-11-17T06:36:21.951 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2023-11-17T06:36:22.034 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2023-11-17T06:36:22.049 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.88.98~20.04.46_amd64.deb ... 2023-11-17T06:36:22.052 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up upower (0.99.11-1build2) ... 2023-11-17T06:36:22.092 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.88.98~20.04.46) ... 2023-11-17T06:36:22.453 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libevdev2:amd64. 2023-11-17T06:36:22.468 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2023-11-17T06:36:22.511 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2023-11-17T06:36:22.573 INFO:teuthology.orchestra.run.smithi193.stdout:upower.service is a disabled or a static unit, not starting it. 2023-11-17T06:36:22.623 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2023-11-17T06:36:22.823 INFO:teuthology.orchestra.run.smithi193.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2023-11-17T06:36:22.841 INFO:teuthology.orchestra.run.smithi193.stdout:Adding system user `usbmux' (UID 114) ... 2023-11-17T06:36:22.842 INFO:teuthology.orchestra.run.smithi193.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2023-11-17T06:36:22.893 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package thermald. 2023-11-17T06:36:22.904 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2023-11-17T06:36:22.948 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2023-11-17T06:36:23.340 INFO:teuthology.orchestra.run.smithi193.stdout:Not creating home directory `/var/lib/usbmux'. 2023-11-17T06:36:23.511 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package upower. 2023-11-17T06:36:23.525 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2023-11-17T06:36:23.557 INFO:teuthology.orchestra.run.smithi193.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2023-11-17T06:36:23.586 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking upower (0.99.11-1build2) ... 2023-11-17T06:36:23.595 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2023-11-17T06:36:23.861 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2023-11-17T06:36:23.861 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2023-11-17T06:36:24.024 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package usbmuxd. 2023-11-17T06:36:24.041 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2023-11-17T06:36:24.082 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2023-11-17T06:36:24.315 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:24.397 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2023-11-17T06:36:24.520 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2023-11-17T06:36:24.646 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2023-11-17T06:36:24.772 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2023-11-17T06:36:24.923 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2023-11-17T06:36:25.049 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2023-11-17T06:36:25.175 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up upower (0.99.11-1build2) ... 2023-11-17T06:36:25.634 INFO:teuthology.orchestra.run.smithi193.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:36:25.634 INFO:teuthology.orchestra.run.smithi193.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-17T06:36:25.717 INFO:teuthology.orchestra.run.smithi097.stdout:upower.service is a disabled or a static unit, not starting it. 2023-11-17T06:36:25.771 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2023-11-17T06:36:25.786 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:25.960 INFO:teuthology.orchestra.run.smithi097.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2023-11-17T06:36:25.968 INFO:teuthology.orchestra.run.smithi097.stdout:Adding system user `usbmux' (UID 114) ... 2023-11-17T06:36:25.968 INFO:teuthology.orchestra.run.smithi097.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2023-11-17T06:36:26.705 INFO:teuthology.orchestra.run.smithi097.stdout:Not creating home directory `/var/lib/usbmux'. 2023-11-17T06:36:26.935 INFO:teuthology.orchestra.run.smithi097.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2023-11-17T06:36:26.977 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2023-11-17T06:36:27.170 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:27.285 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2023-11-17T06:36:27.285 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2023-11-17T06:36:27.731 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:28.546 INFO:teuthology.orchestra.run.smithi193.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.88.98~20.04.46) ... 2023-11-17T06:36:28.688 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-17T06:36:28.953 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-17T06:36:29.040 INFO:teuthology.orchestra.run.smithi097.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:36:29.040 INFO:teuthology.orchestra.run.smithi097.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-17T06:36:29.234 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:29.681 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-11-17T06:36:29.773 INFO:teuthology.orchestra.run.smithi193.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:29.841 INFO:teuthology.orchestra.run.smithi193.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-17T06:36:29.842 INFO:teuthology.orchestra.run.smithi193.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:36:30.618 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:33.046 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.88.98~20.04.46) ... 2023-11-17T06:36:33.380 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-11-17T06:36:33.661 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-11-17T06:36:34.406 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2023-11-17T06:36:34.506 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98~20.04.1) ... 2023-11-17T06:36:34.587 INFO:teuthology.orchestra.run.smithi097.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-17T06:36:34.587 INFO:teuthology.orchestra.run.smithi097.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:36:55.913 INFO:teuthology.orchestra.run.smithi193.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-17T06:36:55.913 INFO:teuthology.orchestra.run.smithi193.stdout:Sourcing file `/etc/default/grub' 2023-11-17T06:36:55.937 INFO:teuthology.orchestra.run.smithi193.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-17T06:36:55.955 INFO:teuthology.orchestra.run.smithi193.stdout:Generating grub configuration file ... 2023-11-17T06:36:56.257 INFO:teuthology.orchestra.run.smithi193.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:36:56.281 INFO:teuthology.orchestra.run.smithi193.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:36:56.654 INFO:teuthology.orchestra.run.smithi193.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-17T06:36:56.666 INFO:teuthology.orchestra.run.smithi193.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-17T06:36:58.761 INFO:teuthology.orchestra.run.smithi193.stdout:done 2023-11-17T06:36:59.163 DEBUG:teuthology.orchestra.run.smithi193:> dpkg -s linux-image-generic-hwe-20.04 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Package: linux-image-generic-hwe-20.04 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Status: install ok installed 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Priority: optional 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Section: kernel 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Installed-Size: 20 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T06:36:59.210 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: amd64 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Source: linux-meta-hwe-5.15 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Version: 5.15.0.88.98~20.04.46 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Recommends: thermald 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout:Description: Generic Linux kernel image 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout: This package will always depend on the latest generic kernel image 2023-11-17T06:36:59.211 INFO:teuthology.orchestra.run.smithi193.stdout: available. 2023-11-17T06:36:59.211 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.15.0-88-generic 2023-11-17T06:36:59.211 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T06:36:59.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T06:36:59.212 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-17T06:36:59.212 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T06:36:59.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-get clean 2023-11-17T06:36:59.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo apt-get update 2023-11-17T06:36:59.440 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-17T06:36:59.444 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-17T06:36:59.464 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-17T06:36:59.548 INFO:teuthology.orchestra.run.smithi193.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-17T06:36:59.975 INFO:teuthology.orchestra.run.smithi097.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-17T06:36:59.976 INFO:teuthology.orchestra.run.smithi097.stdout:Sourcing file `/etc/default/grub' 2023-11-17T06:36:59.978 INFO:teuthology.orchestra.run.smithi097.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-17T06:36:59.996 INFO:teuthology.orchestra.run.smithi097.stdout:Generating grub configuration file ... 2023-11-17T06:37:00.273 INFO:teuthology.orchestra.run.smithi097.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:37:00.298 INFO:teuthology.orchestra.run.smithi097.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:37:00.720 INFO:teuthology.orchestra.run.smithi097.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-17T06:37:00.732 INFO:teuthology.orchestra.run.smithi097.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-17T06:37:01.192 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-17T06:37:01.218 DEBUG:teuthology.orchestra.run.smithi193:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2023-11-17T06:37:01.283 INFO:teuthology.orchestra.run.smithi193.stdout:Reading package lists... 2023-11-17T06:37:01.484 INFO:teuthology.orchestra.run.smithi193.stdout:Building dependency tree... 2023-11-17T06:37:01.485 INFO:teuthology.orchestra.run.smithi193.stdout:Reading state information... 2023-11-17T06:37:01.705 INFO:teuthology.orchestra.run.smithi193.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.88.98~20.04.46). 2023-11-17T06:37:01.705 INFO:teuthology.orchestra.run.smithi193.stdout:0 upgraded, 0 newly installed, 0 to remove and 281 not upgraded. 2023-11-17T06:37:01.706 DEBUG:teuthology.orchestra.run.smithi193:> dpkg -s linux-image-generic-hwe-20.04 2023-11-17T06:37:01.727 INFO:teuthology.orchestra.run.smithi193.stdout:Package: linux-image-generic-hwe-20.04 2023-11-17T06:37:01.728 INFO:teuthology.orchestra.run.smithi193.stdout:Status: install ok installed 2023-11-17T06:37:01.728 INFO:teuthology.orchestra.run.smithi193.stdout:Priority: optional 2023-11-17T06:37:01.728 INFO:teuthology.orchestra.run.smithi193.stdout:Section: kernel 2023-11-17T06:37:01.728 INFO:teuthology.orchestra.run.smithi193.stdout:Installed-Size: 20 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: amd64 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Source: linux-meta-hwe-5.15 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Version: 5.15.0.88.98~20.04.46 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T06:37:01.729 INFO:teuthology.orchestra.run.smithi193.stdout:Recommends: thermald 2023-11-17T06:37:01.730 INFO:teuthology.orchestra.run.smithi193.stdout:Description: Generic Linux kernel image 2023-11-17T06:37:01.730 INFO:teuthology.orchestra.run.smithi193.stdout: This package will always depend on the latest generic kernel image 2023-11-17T06:37:01.730 INFO:teuthology.orchestra.run.smithi193.stdout: available. 2023-11-17T06:37:01.730 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-11-17T06:37:01.734 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.6xAn7auoKz 2023-11-17T06:37:01.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6xAn7auoKz 2023-11-17T06:37:01.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.6xAn7auoKz 2023-11-17T06:37:01.830 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.6xAn7auoKz is 9KB 2023-11-17T06:37:01.843 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.6xAn7auoKz 2023-11-17T06:37:01.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-17T06:37:01.905 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:37:01.905 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-17T06:37:01.905 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-17T06:37:01.940 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-17T06:37:01.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo update-grub 2023-11-17T06:37:02.840 INFO:teuthology.orchestra.run.smithi097.stdout:done 2023-11-17T06:37:03.263 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub' 2023-11-17T06:37:03.269 INFO:teuthology.orchestra.run.smithi193.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-17T06:37:03.280 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2023-11-17T06:37:03.340 DEBUG:teuthology.orchestra.run.smithi097:> dpkg -s linux-image-generic-hwe-20.04 2023-11-17T06:37:03.385 INFO:teuthology.orchestra.run.smithi097.stdout:Package: linux-image-generic-hwe-20.04 2023-11-17T06:37:03.385 INFO:teuthology.orchestra.run.smithi097.stdout:Status: install ok installed 2023-11-17T06:37:03.385 INFO:teuthology.orchestra.run.smithi097.stdout:Priority: optional 2023-11-17T06:37:03.385 INFO:teuthology.orchestra.run.smithi097.stdout:Section: kernel 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Installed-Size: 20 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: amd64 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Source: linux-meta-hwe-5.15 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Version: 5.15.0.88.98~20.04.46 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Recommends: thermald 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout:Description: Generic Linux kernel image 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout: This package will always depend on the latest generic kernel image 2023-11-17T06:37:03.386 INFO:teuthology.orchestra.run.smithi097.stdout: available. 2023-11-17T06:37:03.387 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.15.0-88-generic 2023-11-17T06:37:03.387 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T06:37:03.387 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T06:37:03.387 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-17T06:37:03.387 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T06:37:03.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get clean 2023-11-17T06:37:03.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2023-11-17T06:37:03.513 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:37:03.524 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:37:03.595 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-11-17T06:37:03.713 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-11-17T06:37:03.795 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-11-17T06:37:03.862 INFO:teuthology.orchestra.run.smithi193.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-17T06:37:03.874 INFO:teuthology.orchestra.run.smithi193.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-17T06:37:03.883 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-11-17T06:37:04.883 INFO:teuthology.orchestra.run.smithi193.stderr:done 2023-11-17T06:37:04.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2023-11-17T06:37:05.344 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-11-17T06:37:05.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2023-11-17T06:37:05.442 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-11-17T06:37:05.647 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2023-11-17T06:37:05.647 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2023-11-17T06:37:05.868 INFO:teuthology.orchestra.run.smithi097.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.88.98~20.04.46). 2023-11-17T06:37:05.868 INFO:teuthology.orchestra.run.smithi097.stdout:0 upgraded, 0 newly installed, 0 to remove and 281 not upgraded. 2023-11-17T06:37:05.870 DEBUG:teuthology.orchestra.run.smithi097:> dpkg -s linux-image-generic-hwe-20.04 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Package: linux-image-generic-hwe-20.04 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Status: install ok installed 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Priority: optional 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Section: kernel 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Installed-Size: 20 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Maintainer: Ubuntu Kernel Team 2023-11-17T06:37:05.891 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: amd64 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Source: linux-meta-hwe-5.15 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Version: 5.15.0.88.98~20.04.46 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Recommends: thermald 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout:Description: Generic Linux kernel image 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: This package will always depend on the latest generic kernel image 2023-11-17T06:37:05.892 INFO:teuthology.orchestra.run.smithi097.stdout: available. 2023-11-17T06:37:05.892 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2023-11-17T06:37:05.941 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.VawelYkOBJ 2023-11-17T06:37:05.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VawelYkOBJ 2023-11-17T06:37:05.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.VawelYkOBJ 2023-11-17T06:37:05.988 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.VawelYkOBJ is 9KB 2023-11-17T06:37:06.002 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.VawelYkOBJ 2023-11-17T06:37:06.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-17T06:37:06.066 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:37:06.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-17T06:37:06.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-17T06:37:06.100 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-17T06:37:06.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo update-grub 2023-11-17T06:37:07.433 INFO:teuthology.orchestra.run.smithi097.stderr:Sourcing file `/etc/default/grub' 2023-11-17T06:37:07.439 INFO:teuthology.orchestra.run.smithi097.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-17T06:37:07.455 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2023-11-17T06:37:07.720 INFO:teuthology.orchestra.run.smithi097.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-17T06:37:07.732 INFO:teuthology.orchestra.run.smithi097.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-17T06:37:08.093 INFO:teuthology.orchestra.run.smithi097.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-11-17T06:37:08.105 INFO:teuthology.orchestra.run.smithi097.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-11-17T06:37:09.107 INFO:teuthology.orchestra.run.smithi097.stderr:done 2023-11-17T06:37:09.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2023-11-17T06:37:34.910 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-17T06:37:34.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-17T06:37:34.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:37:39.114 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-17T06:37:39.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-11-17T06:37:39.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:38:13.627 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-11-17T06:38:39.673 DEBUG:teuthology.orchestra.remote:timed out 2023-11-17T06:38:48.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-11-17T06:38:48.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:38:48.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-17T06:38:48.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:38:48.946 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-11-17T06:38:49.440 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-17T06:38:49.440 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-88-generic"... 2023-11-17T06:38:49.440 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-11-17T06:38:49.490 INFO:teuthology.orchestra.run.smithi193.stdout:5.15.0-88-generic 2023-11-17T06:38:49.490 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-17T06:38:49.490 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T06:38:49.490 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-17T06:38:50.491 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-17T06:38:50.492 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T06:38:50.578 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2023-11-17T06:38:50.602 DEBUG:teuthology.parallel:result is None 2023-11-17T06:38:56.586 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-11-17T06:38:57.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-11-17T06:38:57.316 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-88-generic"... 2023-11-17T06:38:57.316 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2023-11-17T06:38:57.321 INFO:teuthology.orchestra.run.smithi097.stdout:5.15.0-88-generic 2023-11-17T06:38:57.321 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-17T06:38:57.321 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T06:38:57.321 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-17T06:38:58.322 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-17T06:38:58.322 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T06:38:58.499 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2023-11-17T06:38:58.527 DEBUG:teuthology.parallel:result is None 2023-11-17T06:38:58.527 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-17T06:38:58.534 INFO:teuthology.task.internal:Creating test directory... 2023-11-17T06:38:58.534 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T06:38:58.537 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T06:38:58.544 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-17T06:38:58.556 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-17T06:38:58.565 INFO:teuthology.task.internal:Creating archive directory... 2023-11-17T06:38:58.565 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T06:38:58.586 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T06:38:58.682 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-17T06:38:58.689 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-17T06:38:58.689 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T06:38:58.692 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T06:38:58.710 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T06:38:58.716 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T06:38:58.810 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T06:38:58.820 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T06:38:58.821 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-17T06:38:58.829 INFO:teuthology.task.internal:Configuring sudo... 2023-11-17T06:38:58.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T06:38:58.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T06:38:58.848 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-17T06:38:58.858 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-17T06:38:58.858 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T06:38:58.890 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T06:38:58.899 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T06:38:58.941 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T06:38:58.990 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:38:58.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T06:38:59.051 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T06:38:59.059 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T06:38:59.107 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:38:59.107 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T06:38:59.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2023-11-17T06:38:59.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2023-11-17T06:38:59.257 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-17T06:38:59.266 INFO:teuthology.task.internal:Starting timer... 2023-11-17T06:38:59.266 INFO:teuthology.run_tasks:Running task pcp... 2023-11-17T06:38:59.281 INFO:teuthology.run_tasks:Running task selinux... 2023-11-17T06:38:59.294 DEBUG:teuthology.task.selinux:Excluding smithi097: OS 'ubuntu' does not support SELinux 2023-11-17T06:38:59.294 DEBUG:teuthology.task.selinux:Excluding smithi193: OS 'ubuntu' does not support SELinux 2023-11-17T06:38:59.294 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-17T06:38:59.294 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-17T06:38:59.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-17T06:38:59.295 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-17T06:38:59.307 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-17T06:38:59.307 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-17T06:38:59.323 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-17T06:38:59.324 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-17T06:45:17.727 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2023-11-17T06:45:17.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-11-17T06:45:17.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:45:17.820 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-11-17T06:45:17.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-11-17T06:45:17.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-17T06:45:17.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T06:45:18.014 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-11-17T06:45:18.132 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-11-17T06:45:18.133 INFO:teuthology.run_tasks:Running task clock... 2023-11-17T06:45:18.144 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-17T06:45:18.144 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T06:45:18.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T06:45:18.147 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T06:45:18.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T06:45:18.176 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-17T06:45:18.176 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Command line: ntpd -gq 2023-11-17T06:45:18.178 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: proto: precision = 0.070 usec (-24) 2023-11-17T06:45:18.179 INFO:teuthology.orchestra.run.smithi097.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T06:45:18.179 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T06:45:18.180 INFO:teuthology.orchestra.run.smithi097.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-17T06:45:18.181 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: restrict ::: KOD does nothing without LIMITED. 2023-11-17T06:45:18.182 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen and drop on 0 v6wildcard [::]:123 2023-11-17T06:45:18.182 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-17T06:45:18.182 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen normally on 2 lo 127.0.0.1:123 2023-11-17T06:45:18.182 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen normally on 3 enp3s0f1 172.21.15.97:123 2023-11-17T06:45:18.182 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen normally on 4 lo [::1]:123 2023-11-17T06:45:18.183 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d805%5]:123 2023-11-17T06:45:18.183 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:18 ntpd[23442]: Listening on routing socket on fd #22 for interface updates 2023-11-17T06:45:18.208 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-11-17T06:45:18.209 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Command line: ntpd -gq 2023-11-17T06:45:18.209 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: proto: precision = 0.100 usec (-23) 2023-11-17T06:45:18.210 INFO:teuthology.orchestra.run.smithi193.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T06:45:18.210 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-17T06:45:18.210 INFO:teuthology.orchestra.run.smithi193.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-17T06:45:18.211 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: restrict ::: KOD does nothing without LIMITED. 2023-11-17T06:45:18.211 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen and drop on 0 v6wildcard [::]:123 2023-11-17T06:45:18.211 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-17T06:45:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen normally on 2 lo 127.0.0.1:123 2023-11-17T06:45:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen normally on 3 enp3s0f1 172.21.15.193:123 2023-11-17T06:45:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen normally on 4 lo [::1]:123 2023-11-17T06:45:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7253%5]:123 2023-11-17T06:45:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:18 ntpd[23433]: Listening on routing socket on fd #22 for interface updates 2023-11-17T06:45:19.176 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:19 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.176 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:19 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:19.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:19.212 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:19.212 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:19.212 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:19.212 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:19 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:20.209 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:20.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:20.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:20.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:20.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:20.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:20 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:21.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:21.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:21 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:22.209 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:22.209 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:22.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:22.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:22.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:22.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:22 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:23.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:23.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:23 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:23.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:23.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:23.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:23.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:23.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:23 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:24.209 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:24.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:24.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:24.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:24.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:24.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:24 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:25.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:25.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:25 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:25.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.210 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:25.211 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:25.212 INFO:teuthology.orchestra.run.smithi193.stdout:17 Nov 06:45:25 ntpd[23433]: ntpd: time slew +0.001486 s 2023-11-17T06:45:25.212 INFO:teuthology.orchestra.run.smithi193.stdout:ntpd: time slew +0.001486s 2023-11-17T06:45:25.212 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-17T06:45:25.213 INFO:teuthology.orchestra.run.smithi193.stderr:17 Nov 06:45:25 ntpd[23433]: can't open /var/log/ntpstats/loopstats.20231117: Permission denied 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================== 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:25.347 INFO:teuthology.orchestra.run.smithi193.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:27.180 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:27.181 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.181 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.181 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:27.182 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:27.183 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:27.183 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:27.184 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:27.184 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:27 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:29.177 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/rawstats.20231117: Permission denied 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/peerstats.20231117: Permission denied 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stdout:17 Nov 06:45:29 ntpd[23442]: ntpd: time slew +0.001269 s 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stdout:ntpd: time slew +0.001269s 2023-11-17T06:45:29.178 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-17T06:45:29.179 INFO:teuthology.orchestra.run.smithi097.stderr:17 Nov 06:45:29 ntpd[23442]: can't open /var/log/ntpstats/loopstats.20231117: Permission denied 2023-11-17T06:45:29.299 INFO:teuthology.orchestra.run.smithi097.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T06:45:29.299 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================== 2023-11-17T06:45:29.299 INFO:teuthology.orchestra.run.smithi097.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:29.300 INFO:teuthology.orchestra.run.smithi097.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:29.300 INFO:teuthology.orchestra.run.smithi097.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:29.300 INFO:teuthology.orchestra.run.smithi097.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-11-17T06:45:29.301 INFO:teuthology.run_tasks:Running task nvme_loop... 2023-11-17T06:45:29.310 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2023-11-17T06:45:29.311 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:45:29.311 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:45:29.354 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-17T06:45:29.355 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 494 Links: 1 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:43:48.339620836 +0000 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:43:48.003628582 +0000 2023-11-17T06:45:29.408 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:43:48.003628582 +0000 2023-11-17T06:45:29.409 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:45:29.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-17T06:45:29.463 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:45:29.464 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:45:29.464 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000392441 s, 1.3 MB/s 2023-11-17T06:45:29.465 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-17T06:45:29.515 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2023-11-17T06:45:29.566 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-17T06:45:29.566 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:29.566 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 501 Links: 1 2023-11-17T06:45:29.567 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:29.567 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:43:48.895608020 +0000 2023-11-17T06:45:29.567 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:43:48.571615489 +0000 2023-11-17T06:45:29.567 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:43:48.571615489 +0000 2023-11-17T06:45:29.567 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:45:29.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-17T06:45:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:45:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:45:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000409838 s, 1.2 MB/s 2023-11-17T06:45:29.622 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-17T06:45:29.673 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 508 Links: 1 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:43:49.123602765 +0000 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:43:49.119602858 +0000 2023-11-17T06:45:29.725 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:43:49.119602858 +0000 2023-11-17T06:45:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:45:29.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-17T06:45:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:45:29.779 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:45:29.780 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000378752 s, 1.4 MB/s 2023-11-17T06:45:29.780 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-17T06:45:29.832 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2023-11-17T06:45:29.882 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-17T06:45:29.883 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:29.883 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 514 Links: 1 2023-11-17T06:45:29.883 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:29.883 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:43:49.919584417 +0000 2023-11-17T06:45:29.883 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:43:49.615591425 +0000 2023-11-17T06:45:29.884 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:43:49.615591425 +0000 2023-11-17T06:45:29.884 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:45:29.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-17T06:45:29.941 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:45:29.942 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:45:29.942 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000414171 s, 1.2 MB/s 2023-11-17T06:45:29.943 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-17T06:45:29.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-11-17T06:45:30.108 INFO:teuthology.orchestra.run.smithi097.stdout:loop 2023-11-17T06:45:30.109 INFO:tasks.nvme_loop:Connecting nvme_loop smithi097:/dev/vg_nvme/lv_1... 2023-11-17T06:45:30.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-11-17T06:45:30.134 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-11-17T06:45:30.159 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/vg_nvme/lv_11 2023-11-17T06:45:30.183 INFO:tasks.nvme_loop:Connecting nvme_loop smithi097:/dev/vg_nvme/lv_2... 2023-11-17T06:45:30.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-11-17T06:45:30.250 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-11-17T06:45:30.274 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/vg_nvme/lv_21 2023-11-17T06:45:30.299 INFO:tasks.nvme_loop:Connecting nvme_loop smithi097:/dev/vg_nvme/lv_3... 2023-11-17T06:45:30.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-11-17T06:45:30.369 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-11-17T06:45:30.393 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/vg_nvme/lv_31 2023-11-17T06:45:30.417 INFO:tasks.nvme_loop:Connecting nvme_loop smithi097:/dev/vg_nvme/lv_4... 2023-11-17T06:45:30.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-11-17T06:45:30.484 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-11-17T06:45:30.509 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/vg_nvme/lv_41 2023-11-17T06:45:30.532 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:45:30.532 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:45:30.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo nvme list 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:Node SN Model Namespace Usage Format FW Rev 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/nvme0n1 PHFT6204013T400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/nvme1n1 c1e833fed63937b5bcff Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/nvme2n1 9dde34b887e30b7fdf21 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/nvme3n1 6200f3985392da14bb33 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:30.640 INFO:teuthology.orchestra.run.smithi097.stdout:/dev/nvme4n1 e9979a13e8359236589a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:30.642 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T06:45:30.642 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:45:30.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/scratch_devs 2023-11-17T06:45:30.698 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:45:30.698 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:45:30.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-17T06:45:30.706 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2023-11-17T06:45:30.756 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-17T06:45:30.756 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 493 Links: 1 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:43:42.688013236 +0000 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:43:42.684013327 +0000 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:43:42.684013327 +0000 2023-11-17T06:45:30.757 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:45:30.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-17T06:45:30.812 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:45:30.812 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:45:30.813 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000419624 s, 1.2 MB/s 2023-11-17T06:45:30.815 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-17T06:45:30.866 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 500 Links: 1 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:43:43.515994140 +0000 2023-11-17T06:45:30.917 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:43:43.212001152 +0000 2023-11-17T06:45:30.918 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:43:43.212001152 +0000 2023-11-17T06:45:30.918 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:45:30.918 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-17T06:45:30.971 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:45:30.972 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:45:30.972 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000396058 s, 1.3 MB/s 2023-11-17T06:45:30.973 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-17T06:45:31.030 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2023-11-17T06:45:31.080 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-17T06:45:31.080 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:31.080 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 508 Links: 1 2023-11-17T06:45:31.081 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:31.081 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:43:44.051981781 +0000 2023-11-17T06:45:31.081 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:43:43.731989160 +0000 2023-11-17T06:45:31.081 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:43:43.731989160 +0000 2023-11-17T06:45:31.081 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:45:31.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-17T06:45:31.136 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:45:31.136 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:45:31.136 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000407685 s, 1.3 MB/s 2023-11-17T06:45:31.137 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-17T06:45:31.190 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2023-11-17T06:45:31.240 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-17T06:45:31.240 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-17T06:45:31.240 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 511 Links: 1 2023-11-17T06:45:31.241 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-17T06:45:31.241 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:43:44.275976614 +0000 2023-11-17T06:45:31.241 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:43:44.271976707 +0000 2023-11-17T06:45:31.241 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:43:44.271976707 +0000 2023-11-17T06:45:31.241 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:45:31.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-17T06:45:31.295 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:45:31.296 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:45:31.296 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000415697 s, 1.2 MB/s 2023-11-17T06:45:31.297 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-17T06:45:31.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2023-11-17T06:45:31.467 INFO:teuthology.orchestra.run.smithi193.stdout:loop 2023-11-17T06:45:31.469 INFO:tasks.nvme_loop:Connecting nvme_loop smithi193:/dev/vg_nvme/lv_1... 2023-11-17T06:45:31.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2023-11-17T06:45:31.491 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-11-17T06:45:31.519 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/vg_nvme/lv_11 2023-11-17T06:45:31.541 INFO:tasks.nvme_loop:Connecting nvme_loop smithi193:/dev/vg_nvme/lv_2... 2023-11-17T06:45:31.541 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2023-11-17T06:45:31.608 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-11-17T06:45:31.634 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/vg_nvme/lv_21 2023-11-17T06:45:31.658 INFO:tasks.nvme_loop:Connecting nvme_loop smithi193:/dev/vg_nvme/lv_3... 2023-11-17T06:45:31.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2023-11-17T06:45:31.722 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-11-17T06:45:31.746 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/vg_nvme/lv_31 2023-11-17T06:45:31.770 INFO:tasks.nvme_loop:Connecting nvme_loop smithi193:/dev/vg_nvme/lv_4... 2023-11-17T06:45:31.771 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2023-11-17T06:45:31.839 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-11-17T06:45:31.860 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/vg_nvme/lv_41 2023-11-17T06:45:31.883 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:45:31.884 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:45:31.932 DEBUG:teuthology.orchestra.run.smithi193:> sudo nvme list 2023-11-17T06:45:31.992 INFO:teuthology.orchestra.run.smithi193.stdout:Node SN Model Namespace Usage Format FW Rev 2023-11-17T06:45:31.992 INFO:teuthology.orchestra.run.smithi193.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2023-11-17T06:45:31.993 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/nvme0n1 PHFT620500HB400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2023-11-17T06:45:31.993 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/nvme1n1 8a764a1b7dfaf71f5124 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:31.993 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/nvme2n1 757d10d3ee3d13b852bd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:31.993 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/nvme3n1 6e45eadb96df34fe7a19 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:31.993 INFO:teuthology.orchestra.run.smithi193.stdout:/dev/nvme4n1 5ef285d79cb74cea0c4f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-8 2023-11-17T06:45:31.994 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T06:45:31.994 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:45:31.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/scratch_devs 2023-11-17T06:45:32.053 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-17T06:45:32.153 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-17T06:45:32.154 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T06:45:32.154 INFO:tasks.cephadm:Cluster fsid is e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:45:32.155 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-17T06:45:32.155 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-17T06:45:32.155 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi097': '172.21.15.97', 'mon.smithi193': '172.21.15.193'} 2023-11-17T06:45:32.155 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-17T06:45:32.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2023-11-17T06:45:32.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2023-11-17T06:45:32.190 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref ed3c8e926b9de93026c8a7c1d86be5d6821b52ac)... 2023-11-17T06:45:32.191 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.385 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Nov 17 06:45 /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.386 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.540 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-rw-r-- 1 ubuntu ubuntu 355729 Nov 17 06:45 /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.540 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.555 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T06:45:32.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2023-11-17T06:45:32.670 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2023-11-17T06:45:32.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2023-11-17T06:45:32.738 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2023-11-17T06:45:32.753 INFO:tasks.cephadm:Writing seed config... 2023-11-17T06:45:32.754 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-17T06:45:32.754 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-17T06:45:32.754 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-17T06:45:32.755 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-17T06:45:32.755 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-17T06:45:32.755 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-17T06:45:32.755 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-17T06:45:32.755 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-17T06:45:32.756 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:45:32.756 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-17T06:45:32.801 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e74de5f6-8514-11ee-8dbf-212e2dc638e7 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-11-17T06:45:32.801 DEBUG:teuthology.orchestra.run.smithi097:mon.smithi097> sudo journalctl -f -n 0 -u ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service 2023-11-17T06:45:32.856 INFO:tasks.cephadm:Bootstrapping... 2023-11-17T06:45:32.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac -v bootstrap --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.97 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:45:33.067 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-11-17T06:45:33.206 INFO:teuthology.orchestra.run.smithi097.stderr:-------------------------------------------------------------------------------- 2023-11-17T06:45:33.206 INFO:teuthology.orchestra.run.smithi097.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', '-v', 'bootstrap', '--fsid', 'e74de5f6-8514-11ee-8dbf-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.97', '--skip-admin-label'] 2023-11-17T06:45:33.207 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying podman|docker is present... 2023-11-17T06:45:33.207 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying lvm2 is present... 2023-11-17T06:45:33.207 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying time synchronization is in place... 2023-11-17T06:45:33.213 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T06:45:33.221 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.228 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-11-17T06:45:33.233 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.241 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: masked 2023-11-17T06:45:33.247 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.253 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-11-17T06:45:33.260 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.266 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: enabled 2023-11-17T06:45:33.272 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: active 2023-11-17T06:45:33.274 INFO:teuthology.orchestra.run.smithi097.stderr:Unit ntp.service is enabled and running 2023-11-17T06:45:33.274 INFO:teuthology.orchestra.run.smithi097.stderr:Repeating the final host check... 2023-11-17T06:45:33.274 INFO:teuthology.orchestra.run.smithi097.stderr:docker (/usr/bin/docker) is present 2023-11-17T06:45:33.274 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl is present 2023-11-17T06:45:33.274 INFO:teuthology.orchestra.run.smithi097.stderr:lvcreate is present 2023-11-17T06:45:33.280 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T06:45:33.286 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.293 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-11-17T06:45:33.298 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.305 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: masked 2023-11-17T06:45:33.311 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.318 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-11-17T06:45:33.323 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2023-11-17T06:45:33.331 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: enabled 2023-11-17T06:45:33.336 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: active 2023-11-17T06:45:33.338 INFO:teuthology.orchestra.run.smithi097.stderr:Unit ntp.service is enabled and running 2023-11-17T06:45:33.339 INFO:teuthology.orchestra.run.smithi097.stderr:Host looks OK 2023-11-17T06:45:33.339 INFO:teuthology.orchestra.run.smithi097.stderr:Cluster fsid: e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:45:33.339 INFO:teuthology.orchestra.run.smithi097.stderr:Acquiring lock 140196952541600 on /run/cephadm/e74de5f6-8514-11ee-8dbf-212e2dc638e7.lock 2023-11-17T06:45:33.339 INFO:teuthology.orchestra.run.smithi097.stderr:Lock 140196952541600 acquired on /run/cephadm/e74de5f6-8514-11ee-8dbf-212e2dc638e7.lock 2023-11-17T06:45:33.340 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying IP 172.21.15.97 port 3300 ... 2023-11-17T06:45:33.340 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying IP 172.21.15.97 port 6789 ... 2023-11-17T06:45:33.340 INFO:teuthology.orchestra.run.smithi097.stderr:Base mon IP(s) is [172.21.15.97:3300, 172.21.15.97:6789], mon addrv is [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2023-11-17T06:45:33.344 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 2023-11-17T06:45:33.345 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.97 metric 100 2023-11-17T06:45:33.345 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T06:45:33.345 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.97 2023-11-17T06:45:33.345 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.97 metric 100 2023-11-17T06:45:33.349 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-17T06:45:33.349 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-17T06:45:33.349 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1792sec pref medium 2023-11-17T06:45:33.354 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-17T06:45:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2023-11-17T06:45:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T06:45:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-17T06:45:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2023-11-17T06:45:33.355 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T06:45:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2023-11-17T06:45:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2023-11-17T06:45:33.357 INFO:teuthology.orchestra.run.smithi097.stderr:Mon IP `172.21.15.97` is in CIDR network `172.21.15.254/32` 2023-11-17T06:45:33.358 INFO:teuthology.orchestra.run.smithi097.stderr:Mon IP `172.21.15.97` is in CIDR network `172.21.15.254/32` 2023-11-17T06:45:33.358 INFO:teuthology.orchestra.run.smithi097.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20', '172.21.15.254/32', '172.21.15.254/32'] 2023-11-17T06:45:33.359 INFO:teuthology.orchestra.run.smithi097.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-17T06:45:33.359 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac... 2023-11-17T06:45:33.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac: Pulling from ceph-ci/ceph 2023-11-17T06:45:33.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 837e66e3ff89: Pulling fs layer 2023-11-17T06:45:33.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 0b35c5e1bfec: Pulling fs layer 2023-11-17T06:45:40.898 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 837e66e3ff89: Verifying Checksum 2023-11-17T06:45:40.898 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 837e66e3ff89: Download complete 2023-11-17T06:45:44.459 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 837e66e3ff89: Pull complete 2023-11-17T06:45:52.609 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 0b35c5e1bfec: Verifying Checksum 2023-11-17T06:45:52.609 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 0b35c5e1bfec: Download complete 2023-11-17T06:46:04.618 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: 0b35c5e1bfec: Pull complete 2023-11-17T06:46:04.760 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: Digest: sha256:f94e7380824ed7d68c996c4c1e29ccce9634d15592a525871e481b74ad1cb87c 2023-11-17T06:46:04.827 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: Status: Downloaded newer image for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T06:46:04.860 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/docker: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T06:46:17.702 INFO:teuthology.orchestra.run.smithi097.stderr:ceph: ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable) 2023-11-17T06:46:18.336 INFO:teuthology.orchestra.run.smithi097.stderr:Ceph version: ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable) 2023-11-17T06:46:18.336 INFO:teuthology.orchestra.run.smithi097.stderr:Extracting ceph user uid/gid from container image... 2023-11-17T06:46:19.120 INFO:teuthology.orchestra.run.smithi097.stderr:stat: 167 167 2023-11-17T06:46:19.754 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial keys... 2023-11-17T06:46:20.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQA8DFdlhk9nHxAAB6hk0rkgzSHhHsRsQurz3g== 2023-11-17T06:46:21.989 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQA9DFdlqqPvOhAADB0msJWMnyJ1c7YKjQFqDQ== 2023-11-17T06:46:23.331 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQA/DFdlhSG7ExAAwKVLYwSJBPzoxyAZIn9WBg== 2023-11-17T06:46:23.938 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial monmap... 2023-11-17T06:46:24.782 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T06:46:24.783 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:24.783 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T06:46:25.417 INFO:teuthology.orchestra.run.smithi097.stderr:monmaptool for smithi097 [v2:172.21.15.97:3300,v1:172.21.15.97:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T06:46:25.417 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: set fsid to e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:25.417 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T06:46:25.417 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:46:25.417 INFO:teuthology.orchestra.run.smithi097.stderr:Creating mon... 2023-11-17T06:46:26.200 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T06:46:26.201 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 1 imported monmap: 2023-11-17T06:46:26.201 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: last_changed 2023-11-17T06:46:24.781618+0000 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: created 2023-11-17T06:46:24.781618+0000 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.smithi097 2023-11-17T06:46:26.202 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.229 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 0 /usr/bin/ceph-mon: set fsid to e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:26.229 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T06:46:26.229 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.229 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T06:46:26.229 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Compile date Nov 14 2023 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: DB SUMMARY 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi097/store.db dir, Total Num: 0, files: 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi097/store.db: 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.error_if_exists: 0 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.create_if_missing: 1 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.env: 0x55da8a1c2080 2023-11-17T06:46:26.230 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.fs: Posix File System 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.info_log: 0x55da8bdb93a0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.statistics: (nil) 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.use_fsync: 0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T06:46:26.231 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.db_log_dir: 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi097/store.db 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T06:46:26.232 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.write_buffer_manager: 0x55da8bdbf980 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T06:46:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.unordered_write: 0 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.row_cache: None 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.wal_filter: None 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T06:46:26.234 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.two_write_queues: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.atomic_flush: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T06:46:26.235 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.max_open_files: -1 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Compression algorithms supported: 2023-11-17T06:46:26.236 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kZSTD supported: 0 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kXpressCompression supported: 0 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kZlibCompression supported: 1 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.197+0000 7f7051a6e880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi097/store.db/MANIFEST-000001 2023-11-17T06:46:26.237 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.merge_operator: 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_filter: None 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55da8bccdd38) 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T06:46:26.238 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_cache: 0x55da8bd04d10 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-17T06:46:26.239 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-17T06:46:26.240 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-17T06:46:26.241 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression: NoCompression 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.num_levels: 7 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T06:46:26.242 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T06:46:26.243 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T06:46:26.244 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T06:46:26.245 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.table_properties_collectors: 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T06:46:26.246 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.bloom_locality: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.ttl: 2592000 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi097/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T06:46:26.247 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.201+0000 7f7051a6e880 4 rocksdb: DB pointer 0x55da8bdcd800 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.205+0000 7f703a8f5700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.205+0000 7f703a8f5700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:26.248 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T06:46:26.249 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:26.250 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:26.251 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: 2023-11-17T06:46:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.205+0000 7f7051a6e880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-17T06:46:26.253 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.205+0000 7f7051a6e880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-17T06:46:26.253 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-mon: debug 2023-11-17T06:46:26.205+0000 7f7051a6e880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi097 for mon.smithi097 2023-11-17T06:46:26.800 INFO:teuthology.orchestra.run.smithi097.stderr:create mon.smithi097 on 2023-11-17T06:46:27.025 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2023-11-17T06:46:27.235 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target -> /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target. 2023-11-17T06:46:27.236 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target -> /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target. 2023-11-17T06:46:27.658 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to reset failed state of unit ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service: Unit ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service not loaded. 2023-11-17T06:46:27.666 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target.wants/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service -> /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@.service. 2023-11-17T06:46:27.878 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld does not appear to be present 2023-11-17T06:46:27.879 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T06:46:27.879 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mon to start... 2023-11-17T06:46:27.879 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mon... 2023-11-17T06:46:29.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:29 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.277288+0000 mon.smithi097 (mon.0) 1 : cluster [INF] mon.smithi097 is new leader, mons smithi097 in quorum (ranks 0) 2023-11-17T06:46:29.653 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: cluster: 2023-11-17T06:46:29.653 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: id: e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:29.653 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-17T06:46:29.653 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:29.653 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: services: 2023-11-17T06:46:29.654 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi097 (age 0.362306s) 2023-11-17T06:46:29.654 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-17T06:46:29.654 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-17T06:46:29.654 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:29.654 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: data: 2023-11-17T06:46:29.657 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-17T06:46:29.657 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-17T06:46:29.657 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-17T06:46:29.657 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: pgs: 2023-11-17T06:46:29.657 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:30.292 INFO:teuthology.orchestra.run.smithi097.stderr:mon is available 2023-11-17T06:46:30.292 INFO:teuthology.orchestra.run.smithi097.stderr:Assimilating anything we can from ceph.conf... 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.287020+0000 mon.smithi097 (mon.0) 2 : cluster [INF] mon.smithi097 is new leader, mons smithi097 in quorum (ranks 0) 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.287502+0000 mon.smithi097 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi097=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.290858+0000 mon.smithi097 (mon.0) 4 : cluster [DBG] fsmap 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.294555+0000 mon.smithi097 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: cluster 2023-11-17T06:46:29.294994+0000 mon.smithi097 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T06:46:30.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:30 smithi097 bash[24404]: audit 2023-11-17T06:46:29.649260+0000 mon.smithi097 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.97:0/383740893' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [global] 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: fsid = e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-17T06:46:31.394 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [mgr] 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [osd] 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-17T06:46:31.395 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-17T06:46:31.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:31 smithi097 bash[24404]: audit 2023-11-17T06:46:31.387655+0000 mon.smithi097 (mon.0) 8 : audit [INF] from='client.? 172.21.15.97:0/1890295699' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T06:46:31.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:31 smithi097 bash[24404]: audit 2023-11-17T06:46:31.391679+0000 mon.smithi097 (mon.0) 9 : audit [INF] from='client.? 172.21.15.97:0/1890295699' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-17T06:46:31.983 INFO:teuthology.orchestra.run.smithi097.stderr:Generating new minimal ceph.conf... 2023-11-17T06:46:33.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:33 smithi097 bash[24404]: audit 2023-11-17T06:46:33.129865+0000 mon.smithi097 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.97:0/2815991449' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:46:33.735 INFO:teuthology.orchestra.run.smithi097.stderr:Restarting the monitor... 2023-11-17T06:46:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:33 smithi097 systemd[1]: Stopping Ceph mon.smithi097 for e74de5f6-8514-11ee-8dbf-212e2dc638e7... 2023-11-17T06:46:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:33 smithi097 bash[24801]: Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-mon.smithi097 2023-11-17T06:46:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:33 smithi097 bash[24404]: debug 2023-11-17T06:46:33.917+0000 7fcf5fc07700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi097 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T06:46:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:33 smithi097 bash[24404]: debug 2023-11-17T06:46:33.917+0000 7fcf5fc07700 -1 mon.smithi097@0(leader) e1 *** Got Signal Terminated *** 2023-11-17T06:46:34.484 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:34 smithi097 bash[24837]: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-mon-smithi097 2023-11-17T06:46:34.681 INFO:teuthology.orchestra.run.smithi097.stderr:Setting public_network to 172.21.0.0/20,172.21.15.254/32 in mon config section 2023-11-17T06:46:34.740 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:34 smithi097 bash[24917]: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-mon-smithi097 2023-11-17T06:46:34.740 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:34 smithi097 systemd[1]: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service: Succeeded. 2023-11-17T06:46:34.740 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:34 smithi097 systemd[1]: Stopped Ceph mon.smithi097 for e74de5f6-8514-11ee-8dbf-212e2dc638e7. 2023-11-17T06:46:34.740 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:34 smithi097 systemd[1]: Started Ceph mon.smithi097 for e74de5f6-8514-11ee-8dbf-212e2dc638e7. 2023-11-17T06:46:36.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.857+0000 7f18096ba880 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T06:46:36.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.857+0000 7f18096ba880 0 ceph version 16.2.14-171-ged3c8e92 (ed3c8e926b9de93026c8a7c1d86be5d6821b52ac) pacific (stable), process ceph-mon, pid 8 2023-11-17T06:46:36.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.857+0000 7f18096ba880 0 pidfile_write: ignore empty --pid-file 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 0 load: jerasure load: lrc load: isa 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Compile date Nov 14 2023 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: DB SUMMARY 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: CURRENT file: CURRENT 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi097/store.db dir, Total Num: 1, files: 000004.sst 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi097/store.db: 000006.log size: 86523 ; 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.error_if_exists: 0 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.create_if_missing: 0 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T06:46:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.env: 0x55ae70bc2080 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.fs: Posix File System 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.info_log: 0x55ae7264a580 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.statistics: (nil) 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.use_fsync: 0 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T06:46:36.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T06:46:36.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T06:46:36.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T06:46:36.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T06:46:36.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T06:46:36.194 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T06:46:36.194 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T06:46:36.194 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.db_log_dir: 2023-11-17T06:46:36.194 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi097/store.db 2023-11-17T06:46:36.194 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.write_buffer_manager: 0x55ae73298480 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T06:46:36.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.unordered_write: 0 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.row_cache: None 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.wal_filter: None 2023-11-17T06:46:36.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.two_write_queues: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.atomic_flush: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T06:46:36.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.861+0000 7f18096ba880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_open_files: -1 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Compression algorithms supported: 2023-11-17T06:46:36.198 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kZSTD supported: 0 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kXpressCompression supported: 0 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kZlibCompression supported: 1 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi097/store.db/MANIFEST-000005 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.merge_operator: 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_filter: None 2023-11-17T06:46:36.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ae725edd30) 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cache_index_and_filter_blocks: 1 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: pin_top_level_index_and_filter: 1 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: index_type: 0 2023-11-17T06:46:36.200 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: data_block_index_type: 0 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: index_shortening: 1 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: data_block_hash_table_util_ratio: 0.750000 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: hash_index_allow_collision: 1 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: checksum: 1 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: no_block_cache: 0 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_cache: 0x55ae72624f10 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_cache_name: BinnedLRUCache 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_cache_options: 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: capacity : 536870912 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: num_shard_bits : 4 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: strict_capacity_limit : 0 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: high_pri_pool_ratio: 0.000 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_cache_compressed: (nil) 2023-11-17T06:46:36.201 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: persistent_cache: (nil) 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_size: 4096 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_size_deviation: 10 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_restart_interval: 16 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: index_block_restart_interval: 1 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: metadata_block_size: 4096 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: partition_filters: 0 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: use_delta_encoding: 1 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: whole_key_filtering: 1 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: verify_compression: 0 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: read_amp_bytes_per_bit: 0 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: format_version: 2 2023-11-17T06:46:36.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: enable_index_compression: 1 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: block_align: 0 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression: NoCompression 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.num_levels: 7 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T06:46:36.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T06:46:36.204 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T06:46:36.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T06:46:36.206 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T06:46:36.206 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T06:46:36.206 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T06:46:36.206 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T06:46:36.207 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.table_properties_collectors: 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.bloom_locality: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T06:46:36.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T06:46:36.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T06:46:36.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.ttl: 2592000 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi097/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700203595867748, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700203595869106, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83396, "table_properties": {"data_size": 81663, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76175, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700203595, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.865+0000 7f18096ba880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700203595869660, "job": 1, "event": "recovery_finished"} 2023-11-17T06:46:36.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.869+0000 7f18096ba880 4 rocksdb: DB pointer 0x55ae726ed800 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.869+0000 7f17f04c0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.869+0000 7f17f04c0700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** DB Stats ** 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T06:46:36.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** Compaction Stats [default] ** 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: L0 2/0 83.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.456 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Sum 2/0 83.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** Compaction Stats [default] ** 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:36.457 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T06:46:36.458 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T06:46:36.458 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T06:46:36.458 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Cumulative compaction: 0.00 GB write, 17.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:36.458 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Interval compaction: 0.00 GB write, 17.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** Compaction Stats [default] ** 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: L0 2/0 83.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Sum 2/0 83.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** Compaction Stats [default] ** 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 91.5 0.00 0.00 1 0.001 0 0 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T06:46:36.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T06:46:36.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T06:46:36.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Cumulative compaction: 0.00 GB write, 16.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:36.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T06:46:36.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T06:46:36.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.869+0000 7f18096ba880 0 starting mon.smithi097 rank 0 at public addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] at bind addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi097 fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.869+0000 7f18096ba880 1 mon.smithi097@-1(???) e1 preinit fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).mds e1 new map 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).mds e1 print_map 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: e1 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T06:46:36.626 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: legacy client fscid: -1 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: No filesystems configured 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 0 mon.smithi097@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: audit 2023-11-17T06:46:33.129865+0000 mon.smithi097 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.97:0/2815991449' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: debug 2023-11-17T06:46:35.873+0000 7f18096ba880 1 mon.smithi097@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cluster 2023-11-17T06:46:35.898724+0000 mon.smithi097 (mon.0) 1 : cluster [INF] mon.smithi097 is new leader, mons smithi097 in quorum (ranks 0) 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cluster 2023-11-17T06:46:35.898804+0000 mon.smithi097 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi097=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} removed_ranks: {} 2023-11-17T06:46:36.627 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cluster 2023-11-17T06:46:35.899256+0000 mon.smithi097 (mon.0) 3 : cluster [DBG] fsmap 2023-11-17T06:46:36.628 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cluster 2023-11-17T06:46:35.899307+0000 mon.smithi097 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T06:46:36.628 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:35 smithi097 bash[24958]: cluster 2023-11-17T06:46:35.899746+0000 mon.smithi097 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T06:46:36.850 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-17T06:46:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:46:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:Creating mgr... 2023-11-17T06:46:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying port 9283 ... 2023-11-17T06:46:37.085 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to reset failed state of unit ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mgr.smithi097.swvonl.service: Unit ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mgr.smithi097.swvonl.service not loaded. 2023-11-17T06:46:37.094 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7.target.wants/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mgr.smithi097.swvonl.service -> /etc/systemd/system/ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@.service. 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld does not appear to be present 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld does not appear to be present 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr to start... 2023-11-17T06:46:37.319 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr... 2023-11-17T06:46:37.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:37 smithi097 bash[24958]: audit 2023-11-17T06:46:36.138406+0000 mon.smithi097 (mon.0) 6 : audit [INF] from='client.? 172.21.15.97:0/4018862480' entity='client.admin' 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "e74de5f6-8514-11ee-8dbf-212e2dc638e7", 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T06:46:38.805 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "smithi097" 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T06:46:38.806 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T06:46:38.807 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T06:46:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T06:46:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T06:46:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T06:46:38.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": false, 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:38.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2023-11-17T06:46:29.289106+0000", 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T06:46:38.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:46:39.022 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:38 smithi097 bash[24958]: audit 2023-11-17T06:46:38.801147+0000 mon.smithi097 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.97:0/238174861' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T06:46:39.467 INFO:teuthology.orchestra.run.smithi097.stderr:mgr not available, waiting (1/15)... 2023-11-17T06:46:42.680 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "e74de5f6-8514-11ee-8dbf-212e2dc638e7", 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T06:46:42.681 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "smithi097" 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 6, 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T06:46:42.682 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T06:46:42.683 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T06:46:42.684 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": false, 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:42.685 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2023-11-17T06:46:29.289106+0000", 2023-11-17T06:46:42.686 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:42.686 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:42.686 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T06:46:42.686 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:46:43.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:42 smithi097 bash[24958]: audit 2023-11-17T06:46:42.674812+0000 mon.smithi097 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.97:0/761713609' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T06:46:43.330 INFO:teuthology.orchestra.run.smithi097.stderr:mgr not available, waiting (2/15)... 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: cluster 2023-11-17T06:46:45.156664+0000 mon.smithi097 (mon.0) 9 : cluster [INF] Activating manager daemon smithi097.swvonl 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: cluster 2023-11-17T06:46:45.162856+0000 mon.smithi097 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi097.swvonl(active, starting, since 0.00643261s) 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.164783+0000 mon.smithi097 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.165052+0000 mon.smithi097 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.165229+0000 mon.smithi097 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T06:46:45.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.165437+0000 mon.smithi097 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:46:45.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.166870+0000 mon.smithi097 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr metadata", "who": "smithi097.swvonl", "id": "smithi097.swvonl"}]: dispatch 2023-11-17T06:46:45.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: cluster 2023-11-17T06:46:45.190507+0000 mon.smithi097 (mon.0) 16 : cluster [INF] Manager daemon smithi097.swvonl is now available 2023-11-17T06:46:45.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:45 smithi097 bash[24958]: audit 2023-11-17T06:46:45.206448+0000 mon.smithi097 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' 2023-11-17T06:46:46.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:46 smithi097 bash[24958]: audit 2023-11-17T06:46:45.216304+0000 mon.smithi097 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/mirror_snapshot_schedule"}]: dispatch 2023-11-17T06:46:46.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:46 smithi097 bash[24958]: audit 2023-11-17T06:46:45.220211+0000 mon.smithi097 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/trash_purge_schedule"}]: dispatch 2023-11-17T06:46:46.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:46 smithi097 bash[24958]: audit 2023-11-17T06:46:45.224573+0000 mon.smithi097 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' 2023-11-17T06:46:46.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:46 smithi097 bash[24958]: audit 2023-11-17T06:46:45.227599+0000 mon.smithi097 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.97:0/528021813' entity='mgr.smithi097.swvonl' 2023-11-17T06:46:46.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:46 smithi097 bash[24958]: cluster 2023-11-17T06:46:46.167682+0000 mon.smithi097 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi097.swvonl(active, since 1.01125s) 2023-11-17T06:46:46.768 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "e74de5f6-8514-11ee-8dbf-212e2dc638e7", 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2023-11-17T06:46:46.769 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "smithi097" 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T06:46:46.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T06:46:46.771 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T06:46:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T06:46:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T06:46:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T06:46:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.775 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2023-11-17T06:46:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2023-11-17T06:46:29.289106+0000", 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T06:46:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:46:47.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:47 smithi097 bash[24958]: audit 2023-11-17T06:46:46.764405+0000 mon.smithi097 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.97:0/1217401704' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T06:46:47.403 INFO:teuthology.orchestra.run.smithi097.stderr:mgr is available 2023-11-17T06:46:48.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:48 smithi097 bash[24958]: cluster 2023-11-17T06:46:47.219029+0000 mon.smithi097 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi097.swvonl(active, since 2s) 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [global] 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: fsid = e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:48.758 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [mgr] 2023-11-17T06:46:48.759 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-17T06:46:48.759 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2023-11-17T06:46:48.759 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [osd] 2023-11-17T06:46:48.759 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-17T06:46:48.760 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-17T06:46:49.384 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling cephadm module... 2023-11-17T06:46:49.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:49 smithi097 bash[24958]: audit 2023-11-17T06:46:48.754783+0000 mon.smithi097 (mon.0) 25 : audit [INF] from='client.? 172.21.15.97:0/1178926144' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T06:46:51.173 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:50 smithi097 bash[24958]: audit 2023-11-17T06:46:50.833595+0000 mon.smithi097 (mon.0) 26 : audit [INF] from='client.? 172.21.15.97:0/1669716359' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-17T06:46:52.262 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:51 smithi097 bash[24958]: audit 2023-11-17T06:46:50.891751+0000 mon.smithi097 (mon.0) 27 : audit [INF] from='client.? 172.21.15.97:0/1669716359' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-17T06:46:52.262 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:51 smithi097 bash[24958]: cluster 2023-11-17T06:46:50.891798+0000 mon.smithi097 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi097.swvonl(active, since 5s) 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "active_name": "smithi097.swvonl", 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T06:46:52.944 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:46:53.186 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:52 smithi097 bash[24958]: audit 2023-11-17T06:46:52.942795+0000 mon.smithi097 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.97:0/827689247' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T06:46:53.637 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for the mgr to restart... 2023-11-17T06:46:53.637 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr epoch 5... 2023-11-17T06:46:57.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: cluster 2023-11-17T06:46:57.468646+0000 mon.smithi097 (mon.0) 30 : cluster [INF] Active manager daemon smithi097.swvonl restarted 2023-11-17T06:46:57.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: cluster 2023-11-17T06:46:57.469212+0000 mon.smithi097 (mon.0) 31 : cluster [INF] Activating manager daemon smithi097.swvonl 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: cluster 2023-11-17T06:46:57.475500+0000 mon.smithi097 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: cluster 2023-11-17T06:46:57.475617+0000 mon.smithi097 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi097.swvonl(active, starting, since 0.0065778s) 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.477227+0000 mon.smithi097 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.478148+0000 mon.smithi097 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr metadata", "who": "smithi097.swvonl", "id": "smithi097.swvonl"}]: dispatch 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.479044+0000 mon.smithi097 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.479352+0000 mon.smithi097 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.479623+0000 mon.smithi097 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: cluster 2023-11-17T06:46:57.482805+0000 mon.smithi097 (mon.0) 39 : cluster [INF] Manager daemon smithi097.swvonl is now available 2023-11-17T06:46:57.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:57 smithi097 bash[24958]: audit 2023-11-17T06:46:57.502426+0000 mon.smithi097 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:46:58.495 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:46:58.495 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-17T06:46:58.495 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T06:46:58.495 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:46:58.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: cephadm 2023-11-17T06:46:57.497088+0000 mgr.smithi097.swvonl (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2023-11-17T06:46:58.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.618922+0000 mon.smithi097 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:46:58.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.621059+0000 mon.smithi097 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:46:58.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.625359+0000 mon.smithi097 (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:46:58.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.637292+0000 mon.smithi097 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:46:58.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.650979+0000 mon.smithi097 (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/mirror_snapshot_schedule"}]: dispatch 2023-11-17T06:46:58.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: audit 2023-11-17T06:46:57.655447+0000 mon.smithi097 (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/trash_purge_schedule"}]: dispatch 2023-11-17T06:46:58.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:58 smithi097 bash[24958]: cluster 2023-11-17T06:46:58.480231+0000 mon.smithi097 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi097.swvonl(active, since 1.01118s) 2023-11-17T06:46:59.195 INFO:teuthology.orchestra.run.smithi097.stderr:mgr epoch 5 is available 2023-11-17T06:46:59.195 INFO:teuthology.orchestra.run.smithi097.stderr:Setting orchestrator backend to cephadm... 2023-11-17T06:46:59.915 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:59 smithi097 bash[24958]: audit 2023-11-17T06:46:58.478345+0000 mgr.smithi097.swvonl (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T06:46:59.916 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:46:59 smithi097 bash[24958]: audit 2023-11-17T06:46:58.493456+0000 mgr.smithi097.swvonl (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T06:47:01.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:00 smithi097 bash[24958]: cluster 2023-11-17T06:46:59.626794+0000 mon.smithi097 (mon.0) 48 : cluster [DBG] mgrmap e8: smithi097.swvonl(active, since 2s) 2023-11-17T06:47:01.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:00 smithi097 bash[24958]: audit 2023-11-17T06:47:00.538943+0000 mon.smithi097 (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:01.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:00 smithi097 bash[24958]: audit 2023-11-17T06:47:00.561802+0000 mon.smithi097 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:47:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:01 smithi097 bash[24958]: audit 2023-11-17T06:47:00.529508+0000 mgr.smithi097.swvonl (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:02.505 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: value unchanged 2023-11-17T06:47:03.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:02 smithi097 bash[24958]: audit 2023-11-17T06:47:02.504387+0000 mgr.smithi097.swvonl (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:03.098 INFO:teuthology.orchestra.run.smithi097.stderr:Generating ssh key... 2023-11-17T06:47:05.988 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:05 smithi097 bash[24958]: audit 2023-11-17T06:47:04.426909+0000 mgr.smithi097.swvonl (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:05.988 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:05 smithi097 bash[24958]: cephadm 2023-11-17T06:47:04.427380+0000 mgr.smithi097.swvonl (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2023-11-17T06:47:05.989 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:05 smithi097 bash[24958]: audit 2023-11-17T06:47:04.625140+0000 mon.smithi097 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:05.989 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:05 smithi097 bash[24958]: audit 2023-11-17T06:47:04.629412+0000 mon.smithi097 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:06.557 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuGgPsJw24E2JYfR/XLmG3gOH2zoJy1vMkB0xcUqLo0kZQTcN+lY1ZVj4B1f1gFYn+Cnp2s4fTto7qEzFACMsS2s8hgHHrBSlo+bFH3ik2HjaBvB1FlHO5XiQHCUXsV31TQYTcb6fi9GeRCMzoxnmuyZfOKDjbo5i8wN7Dms4IPMiQZ96h8kyMAP3g2ghvS3uKUemEhgYTjCDKBZIBy34omq4fiTfpGbM6dcVy8IyKg3XNeEfypCAQQMNmAtKqNlFnZbQIuL1ERM/qQwZi/iyXhEGb2DLBOt1mFUcX16rKt5VrAcULkwzlALK2kO7n+DpJFwRmF95NuPix9QD2D7ivs8p86v+DoZO57U6ucz5gJ0pH33Pmc7Hzmj+zLrISWhzuaEkbOoBe6h+iqD208vFlUVLphfTeAG3vk/hnmV42rl+n7uE9QLzKuVdclQHhxjrthJtn58eHE+GwCZsPuUVjjz2EYJcPjicrOdIl0I7b28icEtCkl3YXlzjKFieOZjk= ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:47:07.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:06 smithi097 bash[24958]: audit 2023-11-17T06:47:06.555624+0000 mgr.smithi097.swvonl (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:07.176 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-17T06:47:07.176 INFO:teuthology.orchestra.run.smithi097.stderr:Adding key to root@localhost authorized_keys... 2023-11-17T06:47:07.176 INFO:teuthology.orchestra.run.smithi097.stderr:Adding host smithi097... 2023-11-17T06:47:09.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:08 smithi097 bash[24958]: audit 2023-11-17T06:47:08.583939+0000 mgr.smithi097.swvonl (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "addr": "172.21.15.97", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.810 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Added host 'smithi097' with addr '172.21.15.97' 2023-11-17T06:47:54.811 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying mon service with default placement... 2023-11-17T06:47:54.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:10 smithi097 bash[24958]: audit 2023-11-17T06:47:09.417767+0000 mon.smithi097 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:10 smithi097 bash[24958]: cephadm 2023-11-17T06:47:09.418183+0000 mgr.smithi097.swvonl (mgr.14116) 10 : cephadm [INF] Added host smithi097 2023-11-17T06:47:54.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:10 smithi097 bash[24958]: audit 2023-11-17T06:47:09.456109+0000 mon.smithi097 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:47:54.812 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-17T06:47:54.812 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying mgr service with default placement... 2023-11-17T06:47:54.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:13 smithi097 bash[24958]: audit 2023-11-17T06:47:12.419640+0000 mgr.smithi097.swvonl (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:13 smithi097 bash[24958]: cephadm 2023-11-17T06:47:12.422706+0000 mgr.smithi097.swvonl (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-17T06:47:54.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:13 smithi097 bash[24958]: audit 2023-11-17T06:47:12.425272+0000 mon.smithi097 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:13 smithi097 bash[24958]: audit 2023-11-17T06:47:12.984660+0000 mon.smithi097 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:13 smithi097 bash[24958]: audit 2023-11-17T06:47:13.227220+0000 mon.smithi097 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-17T06:47:54.813 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying crash service with default placement... 2023-11-17T06:47:54.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:15 smithi097 bash[24958]: audit 2023-11-17T06:47:14.754969+0000 mgr.smithi097.swvonl (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:15 smithi097 bash[24958]: cephadm 2023-11-17T06:47:14.757104+0000 mgr.smithi097.swvonl (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-17T06:47:54.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:15 smithi097 bash[24958]: audit 2023-11-17T06:47:14.760007+0000 mon.smithi097 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.814 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-17T06:47:54.814 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying prometheus service with default placement... 2023-11-17T06:47:54.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:17 smithi097 bash[24958]: audit 2023-11-17T06:47:16.833173+0000 mgr.smithi097.swvonl (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:17 smithi097 bash[24958]: cephadm 2023-11-17T06:47:16.835593+0000 mgr.smithi097.swvonl (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2023-11-17T06:47:54.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:17 smithi097 bash[24958]: audit 2023-11-17T06:47:16.839962+0000 mon.smithi097 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.815 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-17T06:47:54.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:18 smithi097 bash[24958]: cluster 2023-11-17T06:47:17.480896+0000 mgr.smithi097.swvonl (mgr.14116) 17 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:18 smithi097 bash[24958]: audit 2023-11-17T06:47:18.292314+0000 mon.smithi097 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.816 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying grafana service with default placement... 2023-11-17T06:47:54.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:19 smithi097 bash[24958]: audit 2023-11-17T06:47:18.904186+0000 mgr.smithi097.swvonl (mgr.14116) 18 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:19 smithi097 bash[24958]: cephadm 2023-11-17T06:47:18.906380+0000 mgr.smithi097.swvonl (mgr.14116) 19 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-17T06:47:54.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:19 smithi097 bash[24958]: audit 2023-11-17T06:47:18.911254+0000 mon.smithi097 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.817 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-17T06:47:54.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:20 smithi097 bash[24958]: cluster 2023-11-17T06:47:19.481338+0000 mgr.smithi097.swvonl (mgr.14116) 20 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.818 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying node-exporter service with default placement... 2023-11-17T06:47:54.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:22 smithi097 bash[24958]: audit 2023-11-17T06:47:21.042150+0000 mgr.smithi097.swvonl (mgr.14116) 21 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:22 smithi097 bash[24958]: cephadm 2023-11-17T06:47:21.044297+0000 mgr.smithi097.swvonl (mgr.14116) 22 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-17T06:47:54.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:22 smithi097 bash[24958]: audit 2023-11-17T06:47:21.048775+0000 mon.smithi097 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:22 smithi097 bash[24958]: cluster 2023-11-17T06:47:21.481817+0000 mgr.smithi097.swvonl (mgr.14116) 23 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.819 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-17T06:47:54.819 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying alertmanager service with default placement... 2023-11-17T06:47:54.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:24 smithi097 bash[24958]: cluster 2023-11-17T06:47:23.482321+0000 mgr.smithi097.swvonl (mgr.14116) 24 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:24 smithi097 bash[24958]: audit 2023-11-17T06:47:23.573300+0000 mgr.smithi097.swvonl (mgr.14116) 25 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:24 smithi097 bash[24958]: cephadm 2023-11-17T06:47:23.575262+0000 mgr.smithi097.swvonl (mgr.14116) 26 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-17T06:47:54.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:24 smithi097 bash[24958]: audit 2023-11-17T06:47:23.579079+0000 mon.smithi097 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.820 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-17T06:47:54.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:26 smithi097 bash[24958]: cluster 2023-11-17T06:47:25.482788+0000 mgr.smithi097.swvonl (mgr.14116) 27 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:26 smithi097 bash[24958]: audit 2023-11-17T06:47:26.095221+0000 mon.smithi097 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.97:0/2314639244' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:27 smithi097 bash[24958]: audit 2023-11-17T06:47:26.086526+0000 mgr.smithi097.swvonl (mgr.14116) 28 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:27 smithi097 bash[24958]: cephadm 2023-11-17T06:47:26.088515+0000 mgr.smithi097.swvonl (mgr.14116) 29 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-17T06:47:54.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:28 smithi097 bash[24958]: cluster 2023-11-17T06:47:27.483246+0000 mgr.smithi097.swvonl (mgr.14116) 30 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.870 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:28 smithi097 bash[24958]: audit 2023-11-17T06:47:28.138024+0000 mon.smithi097 (mon.0) 65 : audit [INF] from='client.? 172.21.15.97:0/3896802543' entity='client.admin' 2023-11-17T06:47:54.870 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling the dashboard module... 2023-11-17T06:47:54.871 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:30 smithi097 bash[24958]: cluster 2023-11-17T06:47:29.483692+0000 mgr.smithi097.swvonl (mgr.14116) 31 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.871 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:30 smithi097 bash[24958]: audit 2023-11-17T06:47:30.177533+0000 mon.smithi097 (mon.0) 66 : audit [INF] from='client.? 172.21.15.97:0/1374067447' entity='client.admin' 2023-11-17T06:47:54.871 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:32 smithi097 bash[24958]: cluster 2023-11-17T06:47:31.484118+0000 mgr.smithi097.swvonl (mgr.14116) 32 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:47:54.871 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:32 smithi097 bash[24958]: audit 2023-11-17T06:47:32.422991+0000 mon.smithi097 (mon.0) 67 : audit [INF] from='client.? 172.21.15.97:0/3351187201' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-17T06:47:54.872 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:33 smithi097 bash[24958]: audit 2023-11-17T06:47:32.913817+0000 mon.smithi097 (mon.0) 68 : audit [INF] from='client.? 172.21.15.97:0/3351187201' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-17T06:47:54.872 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:33 smithi097 bash[24958]: cluster 2023-11-17T06:47:32.913903+0000 mon.smithi097 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi097.swvonl(active, since 35s) 2023-11-17T06:47:54.873 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:47:54.949 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 9, 2023-11-17T06:47:54.949 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2023-11-17T06:47:54.949 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "active_name": "smithi097.swvonl", 2023-11-17T06:47:54.949 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T06:47:54.949 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:47:54.950 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:35 smithi097 bash[24958]: audit 2023-11-17T06:47:35.259911+0000 mon.smithi097 (mon.0) 70 : audit [DBG] from='client.? 172.21.15.97:0/1252905530' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T06:47:54.950 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for the mgr to restart... 2023-11-17T06:47:54.950 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr epoch 9... 2023-11-17T06:47:54.950 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: cluster 2023-11-17T06:47:39.524632+0000 mon.smithi097 (mon.0) 71 : cluster [INF] Active manager daemon smithi097.swvonl restarted 2023-11-17T06:47:54.950 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: cluster 2023-11-17T06:47:39.525050+0000 mon.smithi097 (mon.0) 72 : cluster [INF] Activating manager daemon smithi097.swvonl 2023-11-17T06:47:54.951 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: cluster 2023-11-17T06:47:39.531299+0000 mon.smithi097 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-17T06:47:54.951 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: cluster 2023-11-17T06:47:39.531517+0000 mon.smithi097 (mon.0) 74 : cluster [DBG] mgrmap e10: smithi097.swvonl(active, starting, since 0.0066341s) 2023-11-17T06:47:54.951 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: audit 2023-11-17T06:47:39.533914+0000 mon.smithi097 (mon.0) 75 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:47:54.953 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: audit 2023-11-17T06:47:39.535123+0000 mon.smithi097 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr metadata", "who": "smithi097.swvonl", "id": "smithi097.swvonl"}]: dispatch 2023-11-17T06:47:54.980 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: audit 2023-11-17T06:47:39.536039+0000 mon.smithi097 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T06:47:54.980 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: audit 2023-11-17T06:47:39.536254+0000 mon.smithi097 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T06:47:54.980 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: audit 2023-11-17T06:47:39.536433+0000 mon.smithi097 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T06:47:54.980 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:39 smithi097 bash[24958]: cluster 2023-11-17T06:47:39.558592+0000 mon.smithi097 (mon.0) 80 : cluster [INF] Manager daemon smithi097.swvonl is now available 2023-11-17T06:47:54.981 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2023-11-17T06:47:54.981 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-11-17T06:47:54.981 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T06:47:54.981 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2023-11-17T06:47:54.981 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:40 smithi097 bash[24958]: audit 2023-11-17T06:47:39.688663+0000 mon.smithi097 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:47:54.981 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:40 smithi097 bash[24958]: audit 2023-11-17T06:47:39.692235+0000 mon.smithi097 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:47:54.981 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:40 smithi097 bash[24958]: audit 2023-11-17T06:47:39.736005+0000 mon.smithi097 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/mirror_snapshot_schedule"}]: dispatch 2023-11-17T06:47:54.981 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:40 smithi097 bash[24958]: audit 2023-11-17T06:47:39.740531+0000 mon.smithi097 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/trash_purge_schedule"}]: dispatch 2023-11-17T06:47:54.981 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:40 smithi097 bash[24958]: cluster 2023-11-17T06:47:40.536309+0000 mon.smithi097 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi097.swvonl(active, since 1.01143s) 2023-11-17T06:47:54.982 INFO:teuthology.orchestra.run.smithi097.stderr:mgr epoch 9 is available 2023-11-17T06:47:54.982 INFO:teuthology.orchestra.run.smithi097.stderr:Generating a dashboard self-signed certificate... 2023-11-17T06:47:54.982 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:41 smithi097 bash[24958]: audit 2023-11-17T06:47:40.535045+0000 mgr.smithi097.swvonl (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T06:47:54.982 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:41 smithi097 bash[24958]: audit 2023-11-17T06:47:40.550607+0000 mgr.smithi097.swvonl (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T06:47:54.991 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-17T06:47:54.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:42 smithi097 bash[24958]: cluster 2023-11-17T06:47:41.899098+0000 mon.smithi097 (mon.0) 86 : cluster [DBG] mgrmap e12: smithi097.swvonl(active, since 2s) 2023-11-17T06:47:54.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:42 smithi097 bash[24958]: audit 2023-11-17T06:47:42.698265+0000 mon.smithi097 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:42 smithi097 bash[24958]: audit 2023-11-17T06:47:42.701252+0000 mon.smithi097 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial admin user... 2023-11-17T06:47:54.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:43 smithi097 bash[24958]: audit 2023-11-17T06:47:42.615606+0000 mgr.smithi097.swvonl (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$8jUuRW65b1FNiJOPPKqCIOBsjOk0klAdK7LsrjLXxTe/xoF8aGpjG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700203665, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-17T06:47:54.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:45 smithi097 bash[24958]: audit 2023-11-17T06:47:44.793048+0000 mgr.smithi097.swvonl (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:47:54.994 INFO:teuthology.orchestra.run.smithi097.stderr:Fetching dashboard port number... 2023-11-17T06:47:54.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:46 smithi097 bash[24958]: audit 2023-11-17T06:47:45.024749+0000 mon.smithi097 (mon.0) 89 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.994 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 8443 2023-11-17T06:47:54.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:47 smithi097 bash[24958]: cluster 2023-11-17T06:47:46.027190+0000 mon.smithi097 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi097.swvonl(active, since 6s) 2023-11-17T06:47:54.995 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:47 smithi097 bash[24958]: audit 2023-11-17T06:47:46.529139+0000 mon.smithi097 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld does not appear to be present 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr:Ceph Dashboard is now available at: 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr: URL: https://smithi097.front.sepia.ceph.com:8443/ 2023-11-17T06:47:54.995 INFO:teuthology.orchestra.run.smithi097.stderr: User: admin 2023-11-17T06:47:54.996 INFO:teuthology.orchestra.run.smithi097.stderr: Password: vkotg647ba 2023-11-17T06:47:54.996 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.996 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling autotune for osd_memory_target 2023-11-17T06:47:54.996 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:48 smithi097 bash[24958]: audit 2023-11-17T06:47:47.134335+0000 mon.smithi097 (mon.0) 92 : audit [DBG] from='client.? 172.21.15.97:0/69506667' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-17T06:47:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2023-11-17T06:47:54.997 INFO:teuthology.orchestra.run.smithi097.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-17T06:47:54.997 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.997 INFO:teuthology.orchestra.run.smithi097.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-17T06:47:54.997 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:Or, if you are only running a single cluster on this host: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: ceph telemetry on 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:For more information see: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T06:47:54.998 INFO:teuthology.orchestra.run.smithi097.stderr:Bootstrap complete. 2023-11-17T06:47:54.999 INFO:tasks.cephadm:Fetching config... 2023-11-17T06:47:54.999 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:47:54.999 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-17T06:47:54.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:52 smithi097 bash[24958]: audit 2023-11-17T06:47:51.816957+0000 mon.smithi097 (mon.0) 93 : audit [INF] from='client.? 172.21.15.97:0/3352040796' entity='client.admin' 2023-11-17T06:47:55.007 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-17T06:47:55.008 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:47:55.008 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-17T06:47:55.061 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-17T06:47:55.061 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:47:55.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/keyring of=/dev/stdout 2023-11-17T06:47:55.117 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-17T06:47:55.117 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:47:55.118 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-17T06:47:55.162 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-17T06:47:55.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuGgPsJw24E2JYfR/XLmG3gOH2zoJy1vMkB0xcUqLo0kZQTcN+lY1ZVj4B1f1gFYn+Cnp2s4fTto7qEzFACMsS2s8hgHHrBSlo+bFH3ik2HjaBvB1FlHO5XiQHCUXsV31TQYTcb6fi9GeRCMzoxnmuyZfOKDjbo5i8wN7Dms4IPMiQZ96h8kyMAP3g2ghvS3uKUemEhgYTjCDKBZIBy34omq4fiTfpGbM6dcVy8IyKg3XNeEfypCAQQMNmAtKqNlFnZbQIuL1ERM/qQwZi/iyXhEGb2DLBOt1mFUcX16rKt5VrAcULkwzlALK2kO7n+DpJFwRmF95NuPix9QD2D7ivs8p86v+DoZO57U6ucz5gJ0pH33Pmc7Hzmj+zLrISWhzuaEkbOoBe6h+iqD208vFlUVLphfTeAG3vk/hnmV42rl+n7uE9QLzKuVdclQHhxjrthJtn58eHE+GwCZsPuUVjjz2EYJcPjicrOdIl0I7b28icEtCkl3YXlzjKFieOZjk= ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T06:47:55.224 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuGgPsJw24E2JYfR/XLmG3gOH2zoJy1vMkB0xcUqLo0kZQTcN+lY1ZVj4B1f1gFYn+Cnp2s4fTto7qEzFACMsS2s8hgHHrBSlo+bFH3ik2HjaBvB1FlHO5XiQHCUXsV31TQYTcb6fi9GeRCMzoxnmuyZfOKDjbo5i8wN7Dms4IPMiQZ96h8kyMAP3g2ghvS3uKUemEhgYTjCDKBZIBy34omq4fiTfpGbM6dcVy8IyKg3XNeEfypCAQQMNmAtKqNlFnZbQIuL1ERM/qQwZi/iyXhEGb2DLBOt1mFUcX16rKt5VrAcULkwzlALK2kO7n+DpJFwRmF95NuPix9QD2D7ivs8p86v+DoZO57U6ucz5gJ0pH33Pmc7Hzmj+zLrISWhzuaEkbOoBe6h+iqD208vFlUVLphfTeAG3vk/hnmV42rl+n7uE9QLzKuVdclQHhxjrthJtn58eHE+GwCZsPuUVjjz2EYJcPjicrOdIl0I7b28icEtCkl3YXlzjKFieOZjk= ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:47:55.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuGgPsJw24E2JYfR/XLmG3gOH2zoJy1vMkB0xcUqLo0kZQTcN+lY1ZVj4B1f1gFYn+Cnp2s4fTto7qEzFACMsS2s8hgHHrBSlo+bFH3ik2HjaBvB1FlHO5XiQHCUXsV31TQYTcb6fi9GeRCMzoxnmuyZfOKDjbo5i8wN7Dms4IPMiQZ96h8kyMAP3g2ghvS3uKUemEhgYTjCDKBZIBy34omq4fiTfpGbM6dcVy8IyKg3XNeEfypCAQQMNmAtKqNlFnZbQIuL1ERM/qQwZi/iyXhEGb2DLBOt1mFUcX16rKt5VrAcULkwzlALK2kO7n+DpJFwRmF95NuPix9QD2D7ivs8p86v+DoZO57U6ucz5gJ0pH33Pmc7Hzmj+zLrISWhzuaEkbOoBe6h+iqD208vFlUVLphfTeAG3vk/hnmV42rl+n7uE9QLzKuVdclQHhxjrthJtn58eHE+GwCZsPuUVjjz2EYJcPjicrOdIl0I7b28icEtCkl3YXlzjKFieOZjk= ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T06:47:55.256 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuGgPsJw24E2JYfR/XLmG3gOH2zoJy1vMkB0xcUqLo0kZQTcN+lY1ZVj4B1f1gFYn+Cnp2s4fTto7qEzFACMsS2s8hgHHrBSlo+bFH3ik2HjaBvB1FlHO5XiQHCUXsV31TQYTcb6fi9GeRCMzoxnmuyZfOKDjbo5i8wN7Dms4IPMiQZ96h8kyMAP3g2ghvS3uKUemEhgYTjCDKBZIBy34omq4fiTfpGbM6dcVy8IyKg3XNeEfypCAQQMNmAtKqNlFnZbQIuL1ERM/qQwZi/iyXhEGb2DLBOt1mFUcX16rKt5VrAcULkwzlALK2kO7n+DpJFwRmF95NuPix9QD2D7ivs8p86v+DoZO57U6ucz5gJ0pH33Pmc7Hzmj+zLrISWhzuaEkbOoBe6h+iqD208vFlUVLphfTeAG3vk/hnmV42rl+n7uE9QLzKuVdclQHhxjrthJtn58eHE+GwCZsPuUVjjz2EYJcPjicrOdIl0I7b28icEtCkl3YXlzjKFieOZjk= ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:47:55.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-17T06:47:57.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:57 smithi097 bash[24958]: audit 2023-11-17T06:47:56.236501+0000 mon.smithi097 (mon.0) 94 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:57.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:57 smithi097 bash[24958]: audit 2023-11-17T06:47:56.549530+0000 mon.smithi097 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:47:57.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:57 smithi097 bash[24958]: audit 2023-11-17T06:47:56.554835+0000 mon.smithi097 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:47:57.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:47:57 smithi097 bash[24958]: cephadm 2023-11-17T06:47:56.570978+0000 mgr.smithi097.swvonl (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi097 on smithi097 2023-11-17T06:48:00.662 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-17T06:48:00.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-17T06:48:01.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:00 smithi097 bash[24958]: cluster 2023-11-17T06:47:59.537919+0000 mgr.smithi097.swvonl (mgr.14156) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:01.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:00 smithi097 bash[24958]: audit 2023-11-17T06:47:59.696523+0000 mon.smithi097 (mon.0) 97 : audit [INF] from='client.? 172.21.15.97:0/1585588423' entity='client.admin' 2023-11-17T06:48:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:02 smithi097 bash[24958]: cluster 2023-11-17T06:48:01.538385+0000 mgr.smithi097.swvonl (mgr.14156) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:05.282 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:04 smithi097 bash[24958]: cluster 2023-11-17T06:48:03.538741+0000 mgr.smithi097.swvonl (mgr.14156) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:06.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:06 smithi097 bash[24958]: audit 2023-11-17T06:48:05.238050+0000 mgr.smithi097.swvonl (mgr.14156) 9 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:48:06.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:06 smithi097 bash[24958]: audit 2023-11-17T06:48:05.242506+0000 mon.smithi097 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:06.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:06 smithi097 bash[24958]: cluster 2023-11-17T06:48:05.539100+0000 mgr.smithi097.swvonl (mgr.14156) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:06.753 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2023-11-17T06:48:06.753 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:48:06.753 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2023-11-17T06:48:06.761 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:48:06.761 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T06:48:06.809 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2023-11-17T06:48:06.809 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch host add smithi193 2023-11-17T06:48:08.976 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:08 smithi097 bash[24958]: cluster 2023-11-17T06:48:07.539555+0000 mgr.smithi097.swvonl (mgr.14156) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: cluster 2023-11-17T06:48:09.539964+0000 mgr.smithi097.swvonl (mgr.14156) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: audit 2023-11-17T06:48:09.928881+0000 mon.smithi097 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: audit 2023-11-17T06:48:09.932038+0000 mon.smithi097 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi097", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: audit 2023-11-17T06:48:09.935233+0000 mon.smithi097 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi097", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-17T06:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: audit 2023-11-17T06:48:09.936279+0000 mon.smithi097 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:10 smithi097 bash[24958]: cephadm 2023-11-17T06:48:09.937256+0000 mgr.smithi097.swvonl (mgr.14156) 13 : cephadm [INF] Deploying daemon crash.smithi097 on smithi097 2023-11-17T06:48:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:12 smithi097 bash[24958]: cluster 2023-11-17T06:48:11.540406+0000 mgr.smithi097.swvonl (mgr.14156) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:12 smithi097 bash[24958]: audit 2023-11-17T06:48:12.852781+0000 mon.smithi097 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:14.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: audit 2023-11-17T06:48:13.039132+0000 mon.smithi097 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: audit 2023-11-17T06:48:13.043336+0000 mon.smithi097 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: audit 2023-11-17T06:48:13.047702+0000 mon.smithi097 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: audit 2023-11-17T06:48:13.048471+0000 mgr.smithi097.swvonl (mgr.14156) 15 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: audit 2023-11-17T06:48:13.051923+0000 mon.smithi097 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: cephadm 2023-11-17T06:48:13.069065+0000 mgr.smithi097.swvonl (mgr.14156) 16 : cephadm [INF] Deploying daemon grafana.smithi097 on smithi097 2023-11-17T06:48:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:14 smithi097 bash[24958]: cluster 2023-11-17T06:48:13.540814+0000 mgr.smithi097.swvonl (mgr.14156) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:16.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:15 smithi097 bash[24958]: audit 2023-11-17T06:48:14.743117+0000 mon.smithi097 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:17.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:16 smithi097 bash[24958]: cluster 2023-11-17T06:48:15.541172+0000 mgr.smithi097.swvonl (mgr.14156) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:19.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:18 smithi097 bash[24958]: cluster 2023-11-17T06:48:17.541538+0000 mgr.smithi097.swvonl (mgr.14156) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:21.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:20 smithi097 bash[24958]: cluster 2023-11-17T06:48:19.541875+0000 mgr.smithi097.swvonl (mgr.14156) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:22 smithi097 bash[24958]: cluster 2023-11-17T06:48:21.542180+0000 mgr.smithi097.swvonl (mgr.14156) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:24 smithi097 bash[24958]: cluster 2023-11-17T06:48:23.542614+0000 mgr.smithi097.swvonl (mgr.14156) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:26 smithi097 bash[24958]: cluster 2023-11-17T06:48:25.542971+0000 mgr.smithi097.swvonl (mgr.14156) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:28 smithi097 bash[24958]: cluster 2023-11-17T06:48:27.543435+0000 mgr.smithi097.swvonl (mgr.14156) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:31.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:30 smithi097 bash[24958]: cluster 2023-11-17T06:48:29.543833+0000 mgr.smithi097.swvonl (mgr.14156) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:33.101 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:32 smithi097 bash[24958]: cluster 2023-11-17T06:48:31.544253+0000 mgr.smithi097.swvonl (mgr.14156) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:33.101 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:32 smithi097 bash[24958]: audit 2023-11-17T06:48:32.880869+0000 mon.smithi097 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:34.146 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:33 smithi097 bash[24958]: cephadm 2023-11-17T06:48:32.885721+0000 mgr.smithi097.swvonl (mgr.14156) 27 : cephadm [INF] Deploying daemon node-exporter.smithi097 on smithi097 2023-11-17T06:48:34.146 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:33 smithi097 bash[24958]: audit 2023-11-17T06:48:33.571304+0000 mon.smithi097 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:34.146 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:33 smithi097 bash[24958]: audit 2023-11-17T06:48:33.574589+0000 mon.smithi097 (mon.0) 111 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-17T06:48:34.953 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:34 smithi097 bash[24958]: cluster 2023-11-17T06:48:33.544622+0000 mgr.smithi097.swvonl (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:48:34.953 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:34 smithi097 bash[24958]: audit 2023-11-17T06:48:34.574951+0000 mon.smithi097 (mon.0) 112 : audit [INF] from='mgr.14156 172.21.15.97:0/1706519423' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-17T06:48:34.953 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:34 smithi097 bash[24958]: cluster 2023-11-17T06:48:34.575054+0000 mon.smithi097 (mon.0) 113 : cluster [DBG] mgrmap e14: smithi097.swvonl(active, since 55s) 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: cluster 2023-11-17T06:48:41.280305+0000 mon.smithi097 (mon.0) 114 : cluster [INF] Active manager daemon smithi097.swvonl restarted 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: cluster 2023-11-17T06:48:41.281033+0000 mon.smithi097 (mon.0) 115 : cluster [INF] Activating manager daemon smithi097.swvonl 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: cluster 2023-11-17T06:48:41.284629+0000 mon.smithi097 (mon.0) 116 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: cluster 2023-11-17T06:48:41.284746+0000 mon.smithi097 (mon.0) 117 : cluster [DBG] mgrmap e15: smithi097.swvonl(active, starting, since 0.00403101s) 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.288448+0000 mon.smithi097 (mon.0) 118 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.289025+0000 mon.smithi097 (mon.0) 119 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr metadata", "who": "smithi097.swvonl", "id": "smithi097.swvonl"}]: dispatch 2023-11-17T06:48:42.030 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.289795+0000 mon.smithi097 (mon.0) 120 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.290014+0000 mon.smithi097 (mon.0) 121 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.290193+0000 mon.smithi097 (mon.0) 122 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: cluster 2023-11-17T06:48:41.313200+0000 mon.smithi097 (mon.0) 123 : cluster [INF] Manager daemon smithi097.swvonl is now available 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.454115+0000 mon.smithi097 (mon.0) 124 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.458405+0000 mon.smithi097 (mon.0) 125 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.477389+0000 mon.smithi097 (mon.0) 126 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.499393+0000 mon.smithi097 (mon.0) 127 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/mirror_snapshot_schedule"}]: dispatch 2023-11-17T06:48:42.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:41 smithi097 bash[24958]: audit 2023-11-17T06:48:41.504388+0000 mon.smithi097 (mon.0) 128 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi097.swvonl/trash_purge_schedule"}]: dispatch 2023-11-17T06:48:43.576 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:43 smithi097 bash[24958]: cluster 2023-11-17T06:48:42.289587+0000 mon.smithi097 (mon.0) 129 : cluster [DBG] mgrmap e16: smithi097.swvonl(active, since 1.00887s) 2023-11-17T06:48:45.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:45 smithi097 bash[24958]: cluster 2023-11-17T06:48:44.290652+0000 mon.smithi097 (mon.0) 130 : cluster [DBG] mgrmap e17: smithi097.swvonl(active, since 3s) 2023-11-17T06:48:48.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.407301+0000 mon.smithi097 (mon.0) 131 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.700257+0000 mon.smithi097 (mon.0) 132 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:48:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.701617+0000 mon.smithi097 (mon.0) 133 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:48:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.702681+0000 mon.smithi097 (mon.0) 134 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:48:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.769304+0000 mon.smithi097 (mon.0) 135 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:48 smithi097 bash[24958]: audit 2023-11-17T06:48:47.776224+0000 mon.smithi097 (mon.0) 136 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:49.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:49 smithi097 bash[24958]: cephadm 2023-11-17T06:48:47.704616+0000 mgr.smithi097.swvonl (mgr.14180) 1 : cephadm [INF] Updating smithi097:/etc/ceph/ceph.conf 2023-11-17T06:48:49.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:49 smithi097 bash[24958]: cephadm 2023-11-17T06:48:47.726157+0000 mgr.smithi097.swvonl (mgr.14180) 2 : cephadm [INF] Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2023-11-17T06:48:49.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:49 smithi097 bash[24958]: cephadm 2023-11-17T06:48:47.798361+0000 mgr.smithi097.swvonl (mgr.14180) 3 : cephadm [INF] Deploying daemon prometheus.smithi097 on smithi097 2023-11-17T06:48:56.021 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:55 smithi097 bash[24958]: audit 2023-11-17T06:48:55.432077+0000 mgr.smithi097.swvonl (mgr.14180) 4 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:48:56.274 INFO:teuthology.orchestra.run.smithi193.stdout:Added host 'smithi193' with addr '172.21.15.193' 2023-11-17T06:48:57.043 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-17T06:48:57.533 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:57 smithi097 bash[24958]: audit 2023-11-17T06:48:56.271678+0000 mon.smithi097 (mon.0) 137 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:48:57.533 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:57 smithi097 bash[24958]: cephadm 2023-11-17T06:48:56.272474+0000 mgr.smithi097.swvonl (mgr.14180) 5 : cephadm [INF] Added host smithi193 2023-11-17T06:48:59.861 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-17T06:48:59.862 INFO:teuthology.orchestra.run.smithi193.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2023-11-17T06:49:00.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:59 smithi097 bash[24958]: audit 2023-11-17T06:48:58.918295+0000 mon.smithi097 (mon.0) 138 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:00.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:59 smithi097 bash[24958]: audit 2023-11-17T06:48:58.920546+0000 mon.smithi097 (mon.0) 139 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:49:00.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:48:59 smithi097 bash[24958]: audit 2023-11-17T06:48:59.462564+0000 mon.smithi097 (mon.0) 140 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:00.587 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-17T06:49:00.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd crush tunables default 2023-11-17T06:49:01.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:00 smithi097 bash[24958]: audit 2023-11-17T06:48:59.858998+0000 mgr.smithi097.swvonl (mgr.14180) 6 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:49:02.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:02 smithi097 bash[24958]: cluster 2023-11-17T06:49:01.291632+0000 mgr.smithi097.swvonl (mgr.14180) 7 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:02.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:02 smithi097 bash[24958]: audit 2023-11-17T06:49:01.482193+0000 mon.smithi097 (mon.0) 141 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:02.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:02 smithi097 bash[24958]: audit 2023-11-17T06:49:01.570510+0000 mon.smithi097 (mon.0) 142 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:04.493 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2023-11-17T06:49:04.778 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:04 smithi097 bash[24958]: cluster 2023-11-17T06:49:03.292238+0000 mgr.smithi097.swvonl (mgr.14180) 8 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:04.778 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:04 smithi097 bash[24958]: audit 2023-11-17T06:49:04.083981+0000 mon.smithi097 (mon.0) 143 : audit [INF] from='client.? 172.21.15.97:0/168726393' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-17T06:49:05.344 INFO:tasks.cephadm:Adding mon.smithi097 on smithi097 2023-11-17T06:49:05.345 INFO:tasks.cephadm:Adding mon.smithi193 on smithi193 2023-11-17T06:49:05.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch apply mon '2;smithi097:172.21.15.97=smithi097;smithi193:172.21.15.193=smithi193' 2023-11-17T06:49:05.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:05 smithi097 bash[24958]: audit 2023-11-17T06:49:04.486387+0000 mon.smithi097 (mon.0) 144 : audit [INF] from='client.? 172.21.15.97:0/168726393' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-17T06:49:05.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:05 smithi097 bash[24958]: cluster 2023-11-17T06:49:04.486449+0000 mon.smithi097 (mon.0) 145 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T06:49:05.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:05 smithi097 bash[24958]: audit 2023-11-17T06:49:04.738864+0000 mon.smithi097 (mon.0) 146 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:06.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:06 smithi097 bash[24958]: cluster 2023-11-17T06:49:05.292708+0000 mgr.smithi097.swvonl (mgr.14180) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.228930+0000 mon.smithi097 (mon.0) 147 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: cluster 2023-11-17T06:49:07.293177+0000 mgr.smithi097.swvonl (mgr.14180) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.600256+0000 mon.smithi097 (mon.0) 148 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.601660+0000 mon.smithi097 (mon.0) 149 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:49:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.603474+0000 mon.smithi097 (mon.0) 150 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:08.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.604916+0000 mon.smithi097 (mon.0) 151 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:49:08.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: cephadm 2023-11-17T06:49:07.607300+0000 mgr.smithi097.swvonl (mgr.14180) 11 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2023-11-17T06:49:08.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.880396+0000 mon.smithi097 (mon.0) 152 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:08.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.887713+0000 mon.smithi097 (mon.0) 153 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:08.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.891038+0000 mon.smithi097 (mon.0) 154 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:49:08.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.897250+0000 mon.smithi097 (mon.0) 155 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-17T06:49:08.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:08 smithi097 bash[24958]: audit 2023-11-17T06:49:07.898667+0000 mon.smithi097 (mon.0) 156 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:08.696 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2023-11-17T06:49:09.511 DEBUG:teuthology.orchestra.run.smithi193:mon.smithi193> sudo journalctl -f -n 0 -u ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi193.service 2023-11-17T06:49:09.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T06:49:09.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T06:49:09.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:09 smithi097 bash[24958]: cephadm 2023-11-17T06:49:07.774837+0000 mgr.smithi097.swvonl (mgr.14180) 12 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2023-11-17T06:49:09.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:09 smithi097 bash[24958]: cephadm 2023-11-17T06:49:07.900026+0000 mgr.smithi097.swvonl (mgr.14180) 13 : cephadm [INF] Deploying daemon crash.smithi193 on smithi193 2023-11-17T06:49:09.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:09 smithi097 bash[24958]: audit 2023-11-17T06:49:08.694742+0000 mon.smithi097 (mon.0) 157 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:10.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:10 smithi097 bash[24958]: audit 2023-11-17T06:49:08.683945+0000 mgr.smithi097.swvonl (mgr.14180) 14 : audit [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi097:172.21.15.97=smithi097;smithi193:172.21.15.193=smithi193", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:49:10.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:10 smithi097 bash[24958]: cephadm 2023-11-17T06:49:08.687962+0000 mgr.smithi097.swvonl (mgr.14180) 15 : cephadm [INF] Saving service mon spec with placement smithi097:172.21.15.97=smithi097;smithi193:172.21.15.193=smithi193;count:2 2023-11-17T06:49:10.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:10 smithi097 bash[24958]: cluster 2023-11-17T06:49:09.293583+0000 mgr.smithi097.swvonl (mgr.14180) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:10.796 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-11-17T06:49:12.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: cluster 2023-11-17T06:49:11.293999+0000 mgr.smithi097.swvonl (mgr.14180) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:12.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: audit 2023-11-17T06:49:11.548367+0000 mon.smithi097 (mon.0) 158 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: audit 2023-11-17T06:49:11.570550+0000 mon.smithi097 (mon.0) 159 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.zqrhit", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T06:49:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: audit 2023-11-17T06:49:11.574633+0000 mon.smithi097 (mon.0) 160 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.zqrhit", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-17T06:49:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: audit 2023-11-17T06:49:11.576041+0000 mon.smithi097 (mon.0) 161 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T06:49:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: audit 2023-11-17T06:49:11.577456+0000 mon.smithi097 (mon.0) 162 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:12 smithi097 bash[24958]: cephadm 2023-11-17T06:49:11.578469+0000 mgr.smithi097.swvonl (mgr.14180) 18 : cephadm [INF] Deploying daemon mgr.smithi193.zqrhit on smithi193 2023-11-17T06:49:13.769 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-17T06:49:13.770 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","modified":"2023-11-17T06:46:24.781618Z","created":"2023-11-17T06:46:24.781618Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi097","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T06:49:13.772 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2023-11-17T06:49:15.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:14 smithi097 bash[24958]: cluster 2023-11-17T06:49:13.294496+0000 mgr.smithi097.swvonl (mgr.14180) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:15.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:14 smithi097 bash[24958]: audit 2023-11-17T06:49:13.768282+0000 mon.smithi097 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.193:0/1322920022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T06:49:15.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T06:49:15.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T06:49:16.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: audit 2023-11-17T06:49:14.710877+0000 mon.smithi097 (mon.0) 164 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: audit 2023-11-17T06:49:14.713414+0000 mon.smithi097 (mon.0) 165 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T06:49:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: audit 2023-11-17T06:49:14.714707+0000 mon.smithi097 (mon.0) 166 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T06:49:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: audit 2023-11-17T06:49:14.716054+0000 mon.smithi097 (mon.0) 167 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: cephadm 2023-11-17T06:49:14.717143+0000 mgr.smithi097.swvonl (mgr.14180) 20 : cephadm [INF] Deploying daemon mon.smithi193 on smithi193 2023-11-17T06:49:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:15 smithi097 bash[24958]: cluster 2023-11-17T06:49:15.294958+0000 mgr.smithi097.swvonl (mgr.14180) 21 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:17.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:17 smithi097 bash[24958]: audit 2023-11-17T06:49:16.490987+0000 mon.smithi097 (mon.0) 168 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:18.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:18 smithi097 bash[24958]: cluster 2023-11-17T06:49:17.295356+0000 mgr.smithi097.swvonl (mgr.14180) 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:20.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:20 smithi097 bash[24958]: cluster 2023-11-17T06:49:19.295850+0000 mgr.smithi097.swvonl (mgr.14180) 23 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:21.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-17T06:49:21.044 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","modified":"2023-11-17T06:46:24.781618Z","created":"2023-11-17T06:46:24.781618Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi097","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T06:49:21.047 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2023-11-17T06:49:22.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: audit 2023-11-17T06:49:20.965408+0000 mon.smithi097 (mon.0) 169 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:22.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: cephadm 2023-11-17T06:49:20.969211+0000 mgr.smithi097.swvonl (mgr.14180) 24 : cephadm [INF] Deploying daemon node-exporter.smithi193 on smithi193 2023-11-17T06:49:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: audit 2023-11-17T06:49:21.042586+0000 mon.smithi097 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.193:0/3317431099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T06:49:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: cluster 2023-11-17T06:49:21.296311+0000 mgr.smithi097.swvonl (mgr.14180) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: audit 2023-11-17T06:49:21.497201+0000 mon.smithi097 (mon.0) 171 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: audit 2023-11-17T06:49:21.726236+0000 mon.smithi097 (mon.0) 172 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:21 smithi097 bash[24958]: audit 2023-11-17T06:49:21.729376+0000 mon.smithi097 (mon.0) 173 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:49:23.177 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:23 smithi193 bash[28559]: debug 2023-11-17T06:49:23.058+0000 7fdd6c5fd700 1 mon.smithi193@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-11-17T06:49:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:22 smithi097 bash[24958]: cluster 2023-11-17T06:49:22.740264+0000 mon.smithi097 (mon.0) 174 : cluster [DBG] Standby manager daemon smithi193.zqrhit started 2023-11-17T06:49:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:22 smithi097 bash[24958]: audit 2023-11-17T06:49:22.746329+0000 mon.smithi097 (mon.0) 175 : audit [DBG] from='mgr.? 172.21.15.193:0/2616375166' entity='mgr.smithi193.zqrhit' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.zqrhit/crt"}]: dispatch 2023-11-17T06:49:23.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:22 smithi097 bash[24958]: audit 2023-11-17T06:49:22.747474+0000 mon.smithi097 (mon.0) 176 : audit [DBG] from='mgr.? 172.21.15.193:0/2616375166' entity='mgr.smithi193.zqrhit' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-17T06:49:23.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:22 smithi097 bash[24958]: audit 2023-11-17T06:49:22.750759+0000 mon.smithi097 (mon.0) 177 : audit [DBG] from='mgr.? 172.21.15.193:0/2616375166' entity='mgr.smithi193.zqrhit' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi193.zqrhit/key"}]: dispatch 2023-11-17T06:49:23.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:22 smithi097 bash[24958]: audit 2023-11-17T06:49:22.751530+0000 mon.smithi097 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.193:0/2616375166' entity='mgr.smithi193.zqrhit' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-17T06:49:23.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T06:49:23.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:23.069550+0000 mon.smithi097 (mon.0) 181 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:23.074149+0000 mon.smithi097 (mon.0) 182 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:23.074524+0000 mon.smithi097 (mon.0) 183 : cluster [INF] mon.smithi097 calling monitor election 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:23.095386+0000 mon.smithi097 (mon.0) 184 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:23.296801+0000 mgr.smithi097.swvonl (mgr.14180) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:24.068295+0000 mon.smithi097 (mon.0) 185 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:25.068348+0000 mon.smithi097 (mon.0) 186 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:25.070580+0000 mon.smithi193 (mon.1) 1 : cluster [INF] mon.smithi193 calling monitor election 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:25.297271+0000 mgr.smithi097.swvonl (mgr.14180) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:26.068632+0000 mon.smithi097 (mon.0) 187 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:27.068982+0000 mon.smithi097 (mon.0) 188 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:27.297698+0000 mgr.smithi097.swvonl (mgr.14180) 28 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:28.068907+0000 mon.smithi097 (mon.0) 189 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.116133+0000 mon.smithi097 (mon.0) 190 : cluster [INF] mon.smithi097 is new leader, mons smithi097,smithi193 in quorum (ranks 0,1) 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.122001+0000 mon.smithi097 (mon.0) 191 : cluster [DBG] monmap e2: 2 mons at {smithi097=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],smithi193=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.122891+0000 mon.smithi097 (mon.0) 192 : cluster [DBG] fsmap 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.122928+0000 mon.smithi097 (mon.0) 193 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.123317+0000 mon.smithi097 (mon.0) 194 : cluster [DBG] mgrmap e18: smithi097.swvonl(active, since 46s), standbys: smithi193.zqrhit 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: cluster 2023-11-17T06:49:28.128871+0000 mon.smithi097 (mon.0) 195 : cluster [INF] overall HEALTH_OK 2023-11-17T06:49:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:28 smithi097 bash[24958]: audit 2023-11-17T06:49:28.133460+0000 mon.smithi097 (mon.0) 196 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:28.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:23.069550+0000 mon.smithi097 (mon.0) 181 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:23.074149+0000 mon.smithi097 (mon.0) 182 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi097"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:23.074524+0000 mon.smithi097 (mon.0) 183 : cluster [INF] mon.smithi097 calling monitor election 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:23.095386+0000 mon.smithi097 (mon.0) 184 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:23.296801+0000 mgr.smithi097.swvonl (mgr.14180) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:24.068295+0000 mon.smithi097 (mon.0) 185 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:25.068348+0000 mon.smithi097 (mon.0) 186 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:25.070580+0000 mon.smithi193 (mon.1) 1 : cluster [INF] mon.smithi193 calling monitor election 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:25.297271+0000 mgr.smithi097.swvonl (mgr.14180) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:26.068632+0000 mon.smithi097 (mon.0) 187 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:27.068982+0000 mon.smithi097 (mon.0) 188 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:27.297698+0000 mgr.smithi097.swvonl (mgr.14180) 28 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:28.068907+0000 mon.smithi097 (mon.0) 189 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.116133+0000 mon.smithi097 (mon.0) 190 : cluster [INF] mon.smithi097 is new leader, mons smithi097,smithi193 in quorum (ranks 0,1) 2023-11-17T06:49:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.122001+0000 mon.smithi097 (mon.0) 191 : cluster [DBG] monmap e2: 2 mons at {smithi097=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],smithi193=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} 2023-11-17T06:49:28.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.122891+0000 mon.smithi097 (mon.0) 192 : cluster [DBG] fsmap 2023-11-17T06:49:28.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.122928+0000 mon.smithi097 (mon.0) 193 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T06:49:28.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.123317+0000 mon.smithi097 (mon.0) 194 : cluster [DBG] mgrmap e18: smithi097.swvonl(active, since 46s), standbys: smithi193.zqrhit 2023-11-17T06:49:28.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: cluster 2023-11-17T06:49:28.128871+0000 mon.smithi097 (mon.0) 195 : cluster [INF] overall HEALTH_OK 2023-11-17T06:49:28.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:28 smithi193 bash[28559]: audit 2023-11-17T06:49:28.133460+0000 mon.smithi097 (mon.0) 196 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:28.735 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-11-17T06:49:28.888 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","modified":"2023-11-17T06:49:23.068921Z","created":"2023-11-17T06:46:24.781618Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi097","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi193","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-17T06:49:28.889 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2023-11-17T06:49:29.339 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-17T06:49:29.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-17T06:49:29.535 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:29 smithi097 bash[24958]: audit 2023-11-17T06:49:28.733847+0000 mon.smithi097 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.193:0/3246166591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T06:49:29.535 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:29 smithi097 bash[24958]: audit 2023-11-17T06:49:29.069250+0000 mon.smithi097 (mon.0) 198 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:29.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:29 smithi193 bash[28559]: audit 2023-11-17T06:49:28.733847+0000 mon.smithi097 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.193:0/3246166591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T06:49:29.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:29 smithi193 bash[28559]: audit 2023-11-17T06:49:29.069250+0000 mon.smithi097 (mon.0) 198 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mon metadata", "id": "smithi193"}]: dispatch 2023-11-17T06:49:30.510 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:30 smithi097 bash[24958]: cluster 2023-11-17T06:49:29.298034+0000 mgr.smithi097.swvonl (mgr.14180) 29 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:30.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:30 smithi193 bash[28559]: cluster 2023-11-17T06:49:29.298034+0000 mgr.smithi097.swvonl (mgr.14180) 29 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:31.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.353848+0000 mon.smithi097 (mon.0) 199 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.355311+0000 mon.smithi097 (mon.0) 200 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:31.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.356753+0000 mon.smithi097 (mon.0) 201 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: cephadm 2023-11-17T06:49:30.358918+0000 mgr.smithi097.swvonl (mgr.14180) 30 : cephadm [INF] Updating smithi097:/etc/ceph/ceph.conf 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: cephadm 2023-11-17T06:49:30.359391+0000 mgr.smithi097.swvonl (mgr.14180) 31 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.434138+0000 mon.smithi097 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.473556+0000 mon.smithi097 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.478569+0000 mon.smithi097 (mon.0) 204 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: cephadm 2023-11-17T06:49:30.483016+0000 mgr.smithi097.swvonl (mgr.14180) 32 : cephadm [INF] Reconfiguring mon.smithi097 (unknown last config time)... 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.483445+0000 mon.smithi097 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.484624+0000 mon.smithi097 (mon.0) 206 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: audit 2023-11-17T06:49:30.485619+0000 mon.smithi097 (mon.0) 207 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:31 smithi097 bash[24958]: cephadm 2023-11-17T06:49:30.486491+0000 mgr.smithi097.swvonl (mgr.14180) 33 : cephadm [INF] Reconfiguring daemon mon.smithi097 on smithi097 2023-11-17T06:49:31.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.353848+0000 mon.smithi097 (mon.0) 199 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.355311+0000 mon.smithi097 (mon.0) 200 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:31.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.356753+0000 mon.smithi097 (mon.0) 201 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: cephadm 2023-11-17T06:49:30.358918+0000 mgr.smithi097.swvonl (mgr.14180) 30 : cephadm [INF] Updating smithi097:/etc/ceph/ceph.conf 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: cephadm 2023-11-17T06:49:30.359391+0000 mgr.smithi097.swvonl (mgr.14180) 31 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.434138+0000 mon.smithi097 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.473556+0000 mon.smithi097 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.478569+0000 mon.smithi097 (mon.0) 204 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: cephadm 2023-11-17T06:49:30.483016+0000 mgr.smithi097.swvonl (mgr.14180) 32 : cephadm [INF] Reconfiguring mon.smithi097 (unknown last config time)... 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.483445+0000 mon.smithi097 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.484624+0000 mon.smithi097 (mon.0) 206 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: audit 2023-11-17T06:49:30.485619+0000 mon.smithi097 (mon.0) 207 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:31 smithi193 bash[28559]: cephadm 2023-11-17T06:49:30.486491+0000 mgr.smithi097.swvonl (mgr.14180) 33 : cephadm [INF] Reconfiguring daemon mon.smithi097 on smithi097 2023-11-17T06:49:32.488 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:32 smithi097 bash[24958]: cluster 2023-11-17T06:49:31.298537+0000 mgr.smithi097.swvonl (mgr.14180) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:32.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:32 smithi193 bash[28559]: cluster 2023-11-17T06:49:31.298537+0000 mgr.smithi097.swvonl (mgr.14180) 34 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:34.072 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:34 smithi097 bash[24958]: cluster 2023-11-17T06:49:33.299022+0000 mgr.smithi097.swvonl (mgr.14180) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:34.073 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:34 smithi097 bash[24958]: audit 2023-11-17T06:49:33.855048+0000 mon.smithi097 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:34.073 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:34 smithi097 bash[24958]: audit 2023-11-17T06:49:33.856954+0000 mon.smithi097 (mon.0) 209 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi097.swvonl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T06:49:34.073 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:34 smithi097 bash[24958]: audit 2023-11-17T06:49:33.858298+0000 mon.smithi097 (mon.0) 210 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T06:49:34.073 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:34 smithi097 bash[24958]: audit 2023-11-17T06:49:33.859314+0000 mon.smithi097 (mon.0) 211 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:34.210 INFO:teuthology.orchestra.run.smithi097.stdout:# minimal ceph.conf for e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:49:34.211 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2023-11-17T06:49:34.211 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = e74de5f6-8514-11ee-8dbf-212e2dc638e7 2023-11-17T06:49:34.211 INFO:teuthology.orchestra.run.smithi097.stdout: mon_host = [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2023-11-17T06:49:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:34 smithi193 bash[28559]: cluster 2023-11-17T06:49:33.299022+0000 mgr.smithi097.swvonl (mgr.14180) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:34 smithi193 bash[28559]: audit 2023-11-17T06:49:33.855048+0000 mon.smithi097 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:34 smithi193 bash[28559]: audit 2023-11-17T06:49:33.856954+0000 mon.smithi097 (mon.0) 209 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi097.swvonl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T06:49:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:34 smithi193 bash[28559]: audit 2023-11-17T06:49:33.858298+0000 mon.smithi097 (mon.0) 210 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T06:49:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:34 smithi193 bash[28559]: audit 2023-11-17T06:49:33.859314+0000 mon.smithi097 (mon.0) 211 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:35 smithi097 bash[24958]: cephadm 2023-11-17T06:49:33.856373+0000 mgr.smithi097.swvonl (mgr.14180) 36 : cephadm [INF] Reconfiguring mgr.smithi097.swvonl (unknown last config time)... 2023-11-17T06:49:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:35 smithi097 bash[24958]: cephadm 2023-11-17T06:49:33.860444+0000 mgr.smithi097.swvonl (mgr.14180) 37 : cephadm [INF] Reconfiguring daemon mgr.smithi097.swvonl on smithi097 2023-11-17T06:49:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:35 smithi097 bash[24958]: audit 2023-11-17T06:49:34.209898+0000 mon.smithi097 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.97:0/960098540' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:35.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:35 smithi193 bash[28559]: cephadm 2023-11-17T06:49:33.856373+0000 mgr.smithi097.swvonl (mgr.14180) 36 : cephadm [INF] Reconfiguring mgr.smithi097.swvonl (unknown last config time)... 2023-11-17T06:49:35.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:35 smithi193 bash[28559]: cephadm 2023-11-17T06:49:33.860444+0000 mgr.smithi097.swvonl (mgr.14180) 37 : cephadm [INF] Reconfiguring daemon mgr.smithi097.swvonl on smithi097 2023-11-17T06:49:35.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:35 smithi193 bash[28559]: audit 2023-11-17T06:49:34.209898+0000 mon.smithi097 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.97:0/960098540' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:35.358 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-17T06:49:35.359 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:49:35.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2023-11-17T06:49:35.371 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:49:35.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T06:49:35.429 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:49:35.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2023-11-17T06:49:35.446 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:49:35.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T06:49:35.506 INFO:tasks.cephadm:Deploying OSDs... 2023-11-17T06:49:35.507 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:49:35.507 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:49:35.512 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T06:49:35.512 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/nvme1n1 2023-11-17T06:49:35.561 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/nvme1n1 2023-11-17T06:49:35.561 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:35.561 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 526 Links: 1 Device type: 103,2 2023-11-17T06:49:35.562 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:35.562 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:47:57.971136281 +0000 2023-11-17T06:49:35.562 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:47:57.843139232 +0000 2023-11-17T06:49:35.562 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:47:57.843139232 +0000 2023-11-17T06:49:35.563 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:49:35.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-11-17T06:49:35.618 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:49:35.618 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:49:35.618 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000441061 s, 1.2 MB/s 2023-11-17T06:49:35.619 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-11-17T06:49:35.673 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/nvme2n1 2023-11-17T06:49:35.721 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/nvme2n1 2023-11-17T06:49:35.721 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:35.721 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 533 Links: 1 Device type: 103,4 2023-11-17T06:49:35.721 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:35.722 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:47:57.995135729 +0000 2023-11-17T06:49:35.722 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:35.722 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:35.722 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:49:35.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-11-17T06:49:35.776 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:49:35.776 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:49:35.776 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000312801 s, 1.6 MB/s 2023-11-17T06:49:35.777 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-11-17T06:49:35.827 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/nvme3n1 2023-11-17T06:49:35.877 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/nvme3n1 2023-11-17T06:49:35.877 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:35.877 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 542 Links: 1 Device type: 103,6 2023-11-17T06:49:35.877 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:35.877 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:47:58.015135267 +0000 2023-11-17T06:49:35.878 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:35.878 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:35.878 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:49:35.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-11-17T06:49:35.929 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:49:35.929 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:49:35.929 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000470562 s, 1.1 MB/s 2023-11-17T06:49:35.930 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-11-17T06:49:35.981 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/nvme4n1 2023-11-17T06:49:36.029 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/nvme4n1 2023-11-17T06:49:36.029 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 550 Links: 1 Device type: 103,8 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-11-17 06:47:58.027134991 +0000 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-11-17 06:47:57.847139140 +0000 2023-11-17T06:49:36.030 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-11-17T06:49:36.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-11-17T06:49:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-11-17T06:49:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-11-17T06:49:36.086 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000487314 s, 1.1 MB/s 2023-11-17T06:49:36.087 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-11-17T06:49:36.139 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:49:36.139 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2023-11-17T06:49:36.147 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2023-11-17T06:49:36.147 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/nvme1n1 2023-11-17T06:49:36.197 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/nvme1n1 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 526 Links: 1 Device type: 103,2 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:49:05.606523653 +0000 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:49:05.506524559 +0000 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:49:05.506524559 +0000 2023-11-17T06:49:36.198 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:49:36.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2023-11-17T06:49:36.253 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:49:36.253 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:49:36.253 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000481749 s, 1.1 MB/s 2023-11-17T06:49:36.254 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2023-11-17T06:49:36.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:36 smithi097 bash[24958]: cluster 2023-11-17T06:49:35.299470+0000 mgr.smithi097.swvonl (mgr.14180) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:36.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:36 smithi193 bash[28559]: cluster 2023-11-17T06:49:35.299470+0000 mgr.smithi097.swvonl (mgr.14180) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:36.307 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/nvme2n1 2023-11-17T06:49:36.357 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/nvme2n1 2023-11-17T06:49:36.357 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 534 Links: 1 Device type: 103,4 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:49:05.622523509 +0000 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:49:05.510524523 +0000 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:49:05.510524523 +0000 2023-11-17T06:49:36.358 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:49:36.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2023-11-17T06:49:36.418 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:49:36.418 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:49:36.418 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000448287 s, 1.1 MB/s 2023-11-17T06:49:36.419 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2023-11-17T06:49:36.471 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/nvme3n1 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/nvme3n1 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 542 Links: 1 Device type: 103,6 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:49:05.634523400 +0000 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:49:05.514524486 +0000 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:49:05.514524486 +0000 2023-11-17T06:49:36.521 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:49:36.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2023-11-17T06:49:36.581 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:49:36.581 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:49:36.581 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000467315 s, 1.1 MB/s 2023-11-17T06:49:36.582 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2023-11-17T06:49:36.635 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/nvme4n1 2023-11-17T06:49:36.685 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/nvme4n1 2023-11-17T06:49:36.685 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2023-11-17T06:49:36.685 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 550 Links: 1 Device type: 103,8 2023-11-17T06:49:36.686 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2023-11-17T06:49:36.686 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-11-17 06:49:05.650523254 +0000 2023-11-17T06:49:36.686 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-11-17 06:49:05.510524523 +0000 2023-11-17T06:49:36.686 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-11-17 06:49:05.510524523 +0000 2023-11-17T06:49:36.686 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-11-17T06:49:36.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2023-11-17T06:49:36.741 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-11-17T06:49:36.741 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-11-17T06:49:36.741 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000477589 s, 1.1 MB/s 2023-11-17T06:49:36.742 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2023-11-17T06:49:36.795 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch apply osd --all-available-devices 2023-11-17T06:49:37.773 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:37 smithi097 bash[24958]: audit 2023-11-17T06:49:36.402334+0000 mon.smithi097 (mon.0) 213 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:37.774 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:37 smithi097 bash[24958]: cephadm 2023-11-17T06:49:36.403732+0000 mgr.smithi097.swvonl (mgr.14180) 39 : cephadm [INF] Reconfiguring alertmanager.smithi097 (dependencies changed)... 2023-11-17T06:49:37.774 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:37 smithi097 bash[24958]: cephadm 2023-11-17T06:49:36.418191+0000 mgr.smithi097.swvonl (mgr.14180) 40 : cephadm [INF] Reconfiguring daemon alertmanager.smithi097 on smithi097 2023-11-17T06:49:37.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:37 smithi193 bash[28559]: audit 2023-11-17T06:49:36.402334+0000 mon.smithi097 (mon.0) 213 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:37.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:37 smithi193 bash[28559]: cephadm 2023-11-17T06:49:36.403732+0000 mgr.smithi097.swvonl (mgr.14180) 39 : cephadm [INF] Reconfiguring alertmanager.smithi097 (dependencies changed)... 2023-11-17T06:49:37.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:37 smithi193 bash[28559]: cephadm 2023-11-17T06:49:36.418191+0000 mgr.smithi097.swvonl (mgr.14180) 40 : cephadm [INF] Reconfiguring daemon alertmanager.smithi097 on smithi097 2023-11-17T06:49:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:38 smithi193 bash[28559]: cluster 2023-11-17T06:49:37.299881+0000 mgr.smithi097.swvonl (mgr.14180) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:38.783 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:38 smithi097 bash[24958]: cluster 2023-11-17T06:49:37.299881+0000 mgr.smithi097.swvonl (mgr.14180) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:39.834 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled osd.all-available-devices update... 2023-11-17T06:49:40.554 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-11-17T06:49:40.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:49:40.682 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:40 smithi097 bash[24958]: cluster 2023-11-17T06:49:39.300225+0000 mgr.smithi097.swvonl (mgr.14180) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:40.682 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:40 smithi097 bash[24958]: audit 2023-11-17T06:49:39.826107+0000 mon.smithi097 (mon.0) 214 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:40.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:40 smithi193 bash[28559]: cluster 2023-11-17T06:49:39.300225+0000 mgr.smithi097.swvonl (mgr.14180) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:40.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:40 smithi193 bash[28559]: audit 2023-11-17T06:49:39.826107+0000 mon.smithi097 (mon.0) 214 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:41.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:41 smithi097 bash[24958]: audit 2023-11-17T06:49:39.816081+0000 mgr.smithi097.swvonl (mgr.14180) 43 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:49:41.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:41 smithi097 bash[24958]: cephadm 2023-11-17T06:49:39.818187+0000 mgr.smithi097.swvonl (mgr.14180) 44 : cephadm [INF] Marking host: smithi097 for OSDSpec preview refresh. 2023-11-17T06:49:41.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:41 smithi097 bash[24958]: cephadm 2023-11-17T06:49:39.818288+0000 mgr.smithi097.swvonl (mgr.14180) 45 : cephadm [INF] Marking host: smithi193 for OSDSpec preview refresh. 2023-11-17T06:49:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:41 smithi097 bash[24958]: cephadm 2023-11-17T06:49:39.818647+0000 mgr.smithi097.swvonl (mgr.14180) 46 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2023-11-17T06:49:41.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:41 smithi193 bash[28559]: audit 2023-11-17T06:49:39.816081+0000 mgr.smithi097.swvonl (mgr.14180) 43 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:49:41.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:41 smithi193 bash[28559]: cephadm 2023-11-17T06:49:39.818187+0000 mgr.smithi097.swvonl (mgr.14180) 44 : cephadm [INF] Marking host: smithi097 for OSDSpec preview refresh. 2023-11-17T06:49:41.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:41 smithi193 bash[28559]: cephadm 2023-11-17T06:49:39.818288+0000 mgr.smithi097.swvonl (mgr.14180) 45 : cephadm [INF] Marking host: smithi193 for OSDSpec preview refresh. 2023-11-17T06:49:41.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:41 smithi193 bash[28559]: cephadm 2023-11-17T06:49:39.818647+0000 mgr.smithi097.swvonl (mgr.14180) 46 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2023-11-17T06:49:42.687 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:42 smithi097 bash[24958]: cluster 2023-11-17T06:49:41.300499+0000 mgr.smithi097.swvonl (mgr.14180) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:42.687 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:42 smithi097 bash[24958]: audit 2023-11-17T06:49:42.132184+0000 mon.smithi097 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:42.687 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:42 smithi097 bash[24958]: audit 2023-11-17T06:49:42.133575+0000 mon.smithi097 (mon.0) 216 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi097", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:49:42.687 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:42 smithi097 bash[24958]: audit 2023-11-17T06:49:42.134618+0000 mon.smithi097 (mon.0) 217 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:42 smithi193 bash[28559]: cluster 2023-11-17T06:49:41.300499+0000 mgr.smithi097.swvonl (mgr.14180) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:42 smithi193 bash[28559]: audit 2023-11-17T06:49:42.132184+0000 mon.smithi097 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:42 smithi193 bash[28559]: audit 2023-11-17T06:49:42.133575+0000 mon.smithi097 (mon.0) 216 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi097", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:49:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:42 smithi193 bash[28559]: audit 2023-11-17T06:49:42.134618+0000 mon.smithi097 (mon.0) 217 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:49:43.714 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:43 smithi097 bash[24958]: cephadm 2023-11-17T06:49:42.133198+0000 mgr.smithi097.swvonl (mgr.14180) 48 : cephadm [INF] Reconfiguring crash.smithi097 (monmap changed)... 2023-11-17T06:49:43.714 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:43 smithi097 bash[24958]: cephadm 2023-11-17T06:49:42.135537+0000 mgr.smithi097.swvonl (mgr.14180) 49 : cephadm [INF] Reconfiguring daemon crash.smithi097 on smithi097 2023-11-17T06:49:43.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:43 smithi193 bash[28559]: cephadm 2023-11-17T06:49:42.133198+0000 mgr.smithi097.swvonl (mgr.14180) 48 : cephadm [INF] Reconfiguring crash.smithi097 (monmap changed)... 2023-11-17T06:49:43.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:43 smithi193 bash[28559]: cephadm 2023-11-17T06:49:42.135537+0000 mgr.smithi097.swvonl (mgr.14180) 49 : cephadm [INF] Reconfiguring daemon crash.smithi097 on smithi097 2023-11-17T06:49:44.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:44 smithi097 bash[24958]: cluster 2023-11-17T06:49:43.300926+0000 mgr.smithi097.swvonl (mgr.14180) 50 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:44.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:44 smithi193 bash[28559]: cluster 2023-11-17T06:49:43.300926+0000 mgr.smithi097.swvonl (mgr.14180) 50 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:46.218 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:49:46.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:46 smithi097 bash[24958]: cluster 2023-11-17T06:49:45.301371+0000 mgr.smithi097.swvonl (mgr.14180) 51 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:46.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:46 smithi097 bash[24958]: audit 2023-11-17T06:49:46.216559+0000 mon.smithi097 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.97:0/903637837' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:49:46.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:46 smithi193 bash[28559]: cluster 2023-11-17T06:49:45.301371+0000 mgr.smithi097.swvonl (mgr.14180) 51 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:46.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:46 smithi193 bash[28559]: audit 2023-11-17T06:49:46.216559+0000 mon.smithi097 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.97:0/903637837' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:49:47.264 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2023-11-17T06:49:47.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:47 smithi097 bash[24958]: audit 2023-11-17T06:49:46.496350+0000 mon.smithi097 (mon.0) 219 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:47.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:47 smithi097 bash[24958]: cephadm 2023-11-17T06:49:46.497668+0000 mgr.smithi097.swvonl (mgr.14180) 52 : cephadm [INF] Reconfiguring grafana.smithi097 (dependencies changed)... 2023-11-17T06:49:47.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:47 smithi097 bash[24958]: cephadm 2023-11-17T06:49:46.540117+0000 mgr.smithi097.swvonl (mgr.14180) 53 : cephadm [INF] Reconfiguring daemon grafana.smithi097 on smithi097 2023-11-17T06:49:47.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:47 smithi193 bash[28559]: audit 2023-11-17T06:49:46.496350+0000 mon.smithi097 (mon.0) 219 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:47.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:47 smithi193 bash[28559]: cephadm 2023-11-17T06:49:46.497668+0000 mgr.smithi097.swvonl (mgr.14180) 52 : cephadm [INF] Reconfiguring grafana.smithi097 (dependencies changed)... 2023-11-17T06:49:47.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:47 smithi193 bash[28559]: cephadm 2023-11-17T06:49:46.540117+0000 mgr.smithi097.swvonl (mgr.14180) 53 : cephadm [INF] Reconfiguring daemon grafana.smithi097 on smithi097 2023-11-17T06:49:48.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:49:48.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:48 smithi097 bash[24958]: cluster 2023-11-17T06:49:47.301766+0000 mgr.smithi097.swvonl (mgr.14180) 54 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:48.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:48 smithi193 bash[28559]: cluster 2023-11-17T06:49:47.301766+0000 mgr.smithi097.swvonl (mgr.14180) 54 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:50.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:50 smithi097 bash[24958]: cluster 2023-11-17T06:49:49.302213+0000 mgr.smithi097.swvonl (mgr.14180) 55 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:50.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:50 smithi193 bash[28559]: cluster 2023-11-17T06:49:49.302213+0000 mgr.smithi097.swvonl (mgr.14180) 55 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:52.214 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:51 smithi097 bash[24958]: cluster 2023-11-17T06:49:51.302616+0000 mgr.smithi097.swvonl (mgr.14180) 56 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:52.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:51 smithi193 bash[28559]: cluster 2023-11-17T06:49:51.302616+0000 mgr.smithi097.swvonl (mgr.14180) 56 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:53.491 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:49:54.472 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:54 smithi097 bash[24958]: audit 2023-11-17T06:49:53.189332+0000 mon.smithi097 (mon.0) 220 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:54.472 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:54 smithi097 bash[24958]: cephadm 2023-11-17T06:49:53.190466+0000 mgr.smithi097.swvonl (mgr.14180) 57 : cephadm [INF] Reconfiguring prometheus.smithi097 (dependencies changed)... 2023-11-17T06:49:54.472 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:54 smithi097 bash[24958]: cephadm 2023-11-17T06:49:53.195487+0000 mgr.smithi097.swvonl (mgr.14180) 58 : cephadm [INF] Reconfiguring daemon prometheus.smithi097 on smithi097 2023-11-17T06:49:54.472 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:54 smithi097 bash[24958]: cluster 2023-11-17T06:49:53.302868+0000 mgr.smithi097.swvonl (mgr.14180) 59 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:54.472 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:54 smithi097 bash[24958]: audit 2023-11-17T06:49:53.490340+0000 mon.smithi097 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.97:0/785982856' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:49:54.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:54 smithi193 bash[28559]: audit 2023-11-17T06:49:53.189332+0000 mon.smithi097 (mon.0) 220 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:49:54.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:54 smithi193 bash[28559]: cephadm 2023-11-17T06:49:53.190466+0000 mgr.smithi097.swvonl (mgr.14180) 57 : cephadm [INF] Reconfiguring prometheus.smithi097 (dependencies changed)... 2023-11-17T06:49:54.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:54 smithi193 bash[28559]: cephadm 2023-11-17T06:49:53.195487+0000 mgr.smithi097.swvonl (mgr.14180) 58 : cephadm [INF] Reconfiguring daemon prometheus.smithi097 on smithi097 2023-11-17T06:49:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:54 smithi193 bash[28559]: cluster 2023-11-17T06:49:53.302868+0000 mgr.smithi097.swvonl (mgr.14180) 59 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:54 smithi193 bash[28559]: audit 2023-11-17T06:49:53.490340+0000 mon.smithi097 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.97:0/785982856' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:49:55.361 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2023-11-17T06:49:56.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:49:56.795 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:56 smithi097 bash[24958]: cluster 2023-11-17T06:49:55.303262+0000 mgr.smithi097.swvonl (mgr.14180) 60 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:56.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:56 smithi193 bash[28559]: cluster 2023-11-17T06:49:55.303262+0000 mgr.smithi097.swvonl (mgr.14180) 60 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:58.728 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:49:58 smithi097 bash[24958]: cluster 2023-11-17T06:49:57.303686+0000 mgr.smithi097.swvonl (mgr.14180) 61 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:49:58.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:49:58 smithi193 bash[28559]: cluster 2023-11-17T06:49:57.303686+0000 mgr.smithi097.swvonl (mgr.14180) 61 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:00.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:00 smithi097 bash[24958]: cluster 2023-11-17T06:49:59.304266+0000 mgr.smithi097.swvonl (mgr.14180) 62 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.692 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:00 smithi097 bash[24958]: cluster 2023-11-17T06:50:00.000128+0000 mon.smithi097 (mon.0) 222 : cluster [INF] overall HEALTH_OK 2023-11-17T06:50:39.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: audit 2023-11-17T06:50:01.276843+0000 mon.smithi097 (mon.0) 223 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: cephadm 2023-11-17T06:50:01.277844+0000 mgr.smithi097.swvonl (mgr.14180) 63 : cephadm [INF] Reconfiguring crash.smithi193 (monmap changed)... 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: audit 2023-11-17T06:50:01.278267+0000 mon.smithi097 (mon.0) 224 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: audit 2023-11-17T06:50:01.279394+0000 mon.smithi097 (mon.0) 225 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: cephadm 2023-11-17T06:50:01.280239+0000 mgr.smithi097.swvonl (mgr.14180) 64 : cephadm [INF] Reconfiguring daemon crash.smithi193 on smithi193 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: cluster 2023-11-17T06:50:01.304583+0000 mgr.smithi097.swvonl (mgr.14180) 65 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.710 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:02 smithi097 bash[24958]: audit 2023-11-17T06:50:01.573124+0000 mon.smithi097 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.97:0/554688455' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:39.712 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:00 smithi193 bash[28559]: cluster 2023-11-17T06:49:59.304266+0000 mgr.smithi097.swvonl (mgr.14180) 62 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:00 smithi193 bash[28559]: cluster 2023-11-17T06:50:00.000128+0000 mon.smithi097 (mon.0) 222 : cluster [INF] overall HEALTH_OK 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: audit 2023-11-17T06:50:01.276843+0000 mon.smithi097 (mon.0) 223 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: cephadm 2023-11-17T06:50:01.277844+0000 mgr.smithi097.swvonl (mgr.14180) 63 : cephadm [INF] Reconfiguring crash.smithi193 (monmap changed)... 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: audit 2023-11-17T06:50:01.278267+0000 mon.smithi097 (mon.0) 224 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi193", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: audit 2023-11-17T06:50:01.279394+0000 mon.smithi097 (mon.0) 225 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: cephadm 2023-11-17T06:50:01.280239+0000 mgr.smithi097.swvonl (mgr.14180) 64 : cephadm [INF] Reconfiguring daemon crash.smithi193 on smithi193 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: cluster 2023-11-17T06:50:01.304583+0000 mgr.smithi097.swvonl (mgr.14180) 65 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:02 smithi193 bash[28559]: audit 2023-11-17T06:50:01.573124+0000 mon.smithi097 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.97:0/554688455' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: audit 2023-11-17T06:50:02.996358+0000 mon.smithi097 (mon.0) 227 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: cephadm 2023-11-17T06:50:02.997334+0000 mgr.smithi097.swvonl (mgr.14180) 66 : cephadm [INF] Reconfiguring mgr.smithi193.zqrhit (monmap changed)... 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: audit 2023-11-17T06:50:02.997734+0000 mon.smithi097 (mon.0) 228 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.zqrhit", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: audit 2023-11-17T06:50:02.998846+0000 mon.smithi097 (mon.0) 229 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: audit 2023-11-17T06:50:02.999651+0000 mon.smithi097 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: cephadm 2023-11-17T06:50:03.000584+0000 mgr.smithi097.swvonl (mgr.14180) 67 : cephadm [INF] Reconfiguring daemon mgr.smithi193.zqrhit on smithi193 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:03 smithi193 bash[28559]: cluster 2023-11-17T06:50:03.304971+0000 mgr.smithi097.swvonl (mgr.14180) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: audit 2023-11-17T06:50:04.730222+0000 mon.smithi097 (mon.0) 231 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: cephadm 2023-11-17T06:50:04.731430+0000 mgr.smithi097.swvonl (mgr.14180) 69 : cephadm [INF] Reconfiguring mon.smithi193 (monmap changed)... 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: audit 2023-11-17T06:50:04.731963+0000 mon.smithi097 (mon.0) 232 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: audit 2023-11-17T06:50:04.733437+0000 mon.smithi097 (mon.0) 233 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: audit 2023-11-17T06:50:04.734944+0000 mon.smithi097 (mon.0) 234 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: cephadm 2023-11-17T06:50:04.736235+0000 mgr.smithi097.swvonl (mgr.14180) 70 : cephadm [INF] Reconfiguring daemon mon.smithi193 on smithi193 2023-11-17T06:50:39.720 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:05 smithi193 bash[28559]: cluster 2023-11-17T06:50:05.305501+0000 mgr.smithi097.swvonl (mgr.14180) 71 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.436258+0000 mon.smithi097 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.441770+0000 mon.smithi097 (mon.0) 236 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.442601+0000 mgr.smithi097.swvonl (mgr.14180) 72 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.444628+0000 mon.smithi097 (mon.0) 237 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.445253+0000 mgr.smithi097.swvonl (mgr.14180) 73 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.451786+0000 mon.smithi097 (mon.0) 238 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.721 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.460696+0000 mon.smithi097 (mon.0) 239 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.461267+0000 mgr.smithi097.swvonl (mgr.14180) 74 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.466558+0000 mon.smithi097 (mon.0) 240 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.467092+0000 mgr.smithi097.swvonl (mgr.14180) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.475668+0000 mon.smithi097 (mon.0) 241 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.481992+0000 mon.smithi097 (mon.0) 242 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.482801+0000 mgr.smithi097.swvonl (mgr.14180) 76 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.488730+0000 mon.smithi097 (mon.0) 243 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.489535+0000 mgr.smithi097.swvonl (mgr.14180) 77 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.496632+0000 mon.smithi097 (mon.0) 244 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:07 smithi193 bash[28559]: audit 2023-11-17T06:50:06.500631+0000 mon.smithi097 (mon.0) 245 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:08 smithi193 bash[28559]: cluster 2023-11-17T06:50:07.305996+0000 mgr.smithi097.swvonl (mgr.14180) 78 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.722 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:10 smithi193 bash[28559]: cluster 2023-11-17T06:50:09.306444+0000 mgr.smithi097.swvonl (mgr.14180) 79 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:11 smithi193 bash[28559]: cluster 2023-11-17T06:50:11.306837+0000 mgr.smithi097.swvonl (mgr.14180) 80 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:11 smithi193 bash[28559]: audit 2023-11-17T06:50:11.834567+0000 mon.smithi097 (mon.0) 246 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.143517+0000 mon.smithi097 (mon.0) 247 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.462485+0000 mon.smithi097 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.739697+0000 mon.smithi097 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.740627+0000 mon.smithi097 (mon.0) 250 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.741601+0000 mon.smithi097 (mon.0) 251 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.748405+0000 mon.smithi097 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.753903+0000 mon.smithi097 (mon.0) 253 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.760088+0000 mon.smithi097 (mon.0) 254 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.763492+0000 mon.smithi097 (mon.0) 255 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.763906+0000 mon.smithi097 (mon.0) 256 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.723 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:13 smithi193 bash[28559]: audit 2023-11-17T06:50:12.764807+0000 mon.smithi097 (mon.0) 257 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:14 smithi193 bash[28559]: cluster 2023-11-17T06:50:13.307291+0000 mgr.smithi097.swvonl (mgr.14180) 81 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:16 smithi193 bash[28559]: cluster 2023-11-17T06:50:15.307752+0000 mgr.smithi097.swvonl (mgr.14180) 82 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:18 smithi193 bash[28559]: cluster 2023-11-17T06:50:17.308175+0000 mgr.smithi097.swvonl (mgr.14180) 83 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:20 smithi193 bash[28559]: cluster 2023-11-17T06:50:19.308622+0000 mgr.smithi097.swvonl (mgr.14180) 84 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: cluster 2023-11-17T06:50:21.309082+0000 mgr.smithi097.swvonl (mgr.14180) 85 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.375232+0000 mon.smithi193 (mon.1) 2 : audit [INF] from='client.? 172.21.15.193:0/1809673918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]: dispatch 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.376511+0000 mon.smithi097 (mon.0) 258 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]: dispatch 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.383925+0000 mon.smithi097 (mon.0) 259 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]': finished 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: cluster 2023-11-17T06:50:21.384075+0000 mon.smithi097 (mon.0) 260 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.384359+0000 mon.smithi097 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.902120+0000 mon.smithi097 (mon.0) 262 : audit [INF] from='client.? 172.21.15.97:0/3308878968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe"}]: dispatch 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.909429+0000 mon.smithi097 (mon.0) 263 : audit [INF] from='client.? 172.21.15.97:0/3308878968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe"}]': finished 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: cluster 2023-11-17T06:50:21.909523+0000 mon.smithi097 (mon.0) 264 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2023-11-17T06:50:39.724 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.909756+0000 mon.smithi097 (mon.0) 265 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:21 smithi193 bash[28559]: audit 2023-11-17T06:50:21.910523+0000 mon.smithi097 (mon.0) 266 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:23 smithi193 bash[28559]: audit 2023-11-17T06:50:22.962079+0000 mon.smithi193 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.193:0/2451937437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:24 smithi193 bash[28559]: cluster 2023-11-17T06:50:23.309438+0000 mgr.smithi097.swvonl (mgr.14180) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:24 smithi193 bash[28559]: audit 2023-11-17T06:50:23.464213+0000 mon.smithi097 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.97:0/1544928080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: cluster 2023-11-17T06:50:25.309870+0000 mgr.smithi097.swvonl (mgr.14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.759223+0000 mon.smithi193 (mon.1) 4 : audit [INF] from='client.? 172.21.15.193:0/2642303979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.760344+0000 mon.smithi097 (mon.0) 268 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]: dispatch 2023-11-17T06:50:39.725 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.770166+0000 mon.smithi097 (mon.0) 269 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]': finished 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: cluster 2023-11-17T06:50:25.770311+0000 mon.smithi097 (mon.0) 270 : cluster [DBG] osdmap e8: 3 total, 0 up, 3 in 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.770504+0000 mon.smithi097 (mon.0) 271 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.771125+0000 mon.smithi097 (mon.0) 272 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:25.771451+0000 mon.smithi097 (mon.0) 273 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.253579+0000 mon.smithi097 (mon.0) 274 : audit [INF] from='client.? 172.21.15.97:0/3420491407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bb2d027-be64-4f25-92eb-0d72ee3f968c"}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.259593+0000 mon.smithi097 (mon.0) 275 : audit [INF] from='client.? 172.21.15.97:0/3420491407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bb2d027-be64-4f25-92eb-0d72ee3f968c"}]': finished 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: cluster 2023-11-17T06:50:26.259642+0000 mon.smithi097 (mon.0) 276 : cluster [DBG] osdmap e9: 4 total, 0 up, 4 in 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.259727+0000 mon.smithi097 (mon.0) 277 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.259969+0000 mon.smithi097 (mon.0) 278 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.726 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.260128+0000 mon.smithi097 (mon.0) 279 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:26 smithi193 bash[28559]: audit 2023-11-17T06:50:26.260378+0000 mon.smithi097 (mon.0) 280 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:27 smithi193 bash[28559]: audit 2023-11-17T06:50:27.314193+0000 mon.smithi193 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.193:0/2197922542' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:28 smithi193 bash[28559]: cluster 2023-11-17T06:50:27.310369+0000 mgr.smithi097.swvonl (mgr.14180) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:28 smithi193 bash[28559]: audit 2023-11-17T06:50:27.792075+0000 mon.smithi097 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.97:0/2838779727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: cluster 2023-11-17T06:50:29.310817+0000 mgr.smithi097.swvonl (mgr.14180) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.108471+0000 mon.smithi193 (mon.1) 6 : audit [INF] from='client.? 172.21.15.193:0/882179693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.109629+0000 mon.smithi097 (mon.0) 282 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.120211+0000 mon.smithi097 (mon.0) 283 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]': finished 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: cluster 2023-11-17T06:50:30.120335+0000 mon.smithi097 (mon.0) 284 : cluster [DBG] osdmap e10: 5 total, 0 up, 5 in 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.120612+0000 mon.smithi097 (mon.0) 285 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.121256+0000 mon.smithi097 (mon.0) 286 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.727 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.121658+0000 mon.smithi097 (mon.0) 287 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.122060+0000 mon.smithi097 (mon.0) 288 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:30 smithi193 bash[28559]: audit 2023-11-17T06:50:30.122471+0000 mon.smithi097 (mon.0) 289 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.575312+0000 mon.smithi097 (mon.0) 290 : audit [INF] from='client.? 172.21.15.97:0/270524419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7463f212-d4f5-427f-a4db-3835fce07e46"}]: dispatch 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.580805+0000 mon.smithi097 (mon.0) 291 : audit [INF] from='client.? 172.21.15.97:0/270524419' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7463f212-d4f5-427f-a4db-3835fce07e46"}]': finished 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: cluster 2023-11-17T06:50:30.580843+0000 mon.smithi097 (mon.0) 292 : cluster [DBG] osdmap e11: 6 total, 0 up, 6 in 2023-11-17T06:50:39.728 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.581002+0000 mon.smithi097 (mon.0) 293 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.763 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.581605+0000 mon.smithi097 (mon.0) 294 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.582065+0000 mon.smithi097 (mon.0) 295 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.582552+0000 mon.smithi097 (mon.0) 296 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.583027+0000 mon.smithi097 (mon.0) 297 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:31 smithi193 bash[28559]: audit 2023-11-17T06:50:30.583435+0000 mon.smithi097 (mon.0) 298 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:32 smithi193 bash[28559]: cluster 2023-11-17T06:50:31.311194+0000 mgr.smithi097.swvonl (mgr.14180) 90 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:32 smithi193 bash[28559]: audit 2023-11-17T06:50:31.729490+0000 mon.smithi193 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.193:0/3468112770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:32 smithi193 bash[28559]: audit 2023-11-17T06:50:32.117927+0000 mon.smithi097 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.97:0/2965368618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:34 smithi193 bash[28559]: cluster 2023-11-17T06:50:33.311680+0000 mgr.smithi097.swvonl (mgr.14180) 91 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.538818+0000 mon.smithi193 (mon.1) 8 : audit [INF] from='client.? 172.21.15.193:0/3242142143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.540014+0000 mon.smithi097 (mon.0) 300 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]: dispatch 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.546732+0000 mon.smithi097 (mon.0) 301 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]': finished 2023-11-17T06:50:39.764 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: cluster 2023-11-17T06:50:34.546847+0000 mon.smithi097 (mon.0) 302 : cluster [DBG] osdmap e12: 7 total, 0 up, 7 in 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.547051+0000 mon.smithi097 (mon.0) 303 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.547817+0000 mon.smithi097 (mon.0) 304 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.548232+0000 mon.smithi097 (mon.0) 305 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.548599+0000 mon.smithi097 (mon.0) 306 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.548965+0000 mon.smithi097 (mon.0) 307 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.549326+0000 mon.smithi097 (mon.0) 308 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.549688+0000 mon.smithi097 (mon.0) 309 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.924378+0000 mon.smithi097 (mon.0) 310 : audit [INF] from='client.? 172.21.15.97:0/226756965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4c1b9cb-3792-496d-b445-63a1e16c3823"}]: dispatch 2023-11-17T06:50:39.765 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.932510+0000 mon.smithi097 (mon.0) 311 : audit [INF] from='client.? 172.21.15.97:0/226756965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4c1b9cb-3792-496d-b445-63a1e16c3823"}]': finished 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: cluster 2023-11-17T06:50:34.932674+0000 mon.smithi097 (mon.0) 312 : cluster [DBG] osdmap e13: 8 total, 0 up, 8 in 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.932835+0000 mon.smithi097 (mon.0) 313 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933089+0000 mon.smithi097 (mon.0) 314 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933293+0000 mon.smithi097 (mon.0) 315 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933458+0000 mon.smithi097 (mon.0) 316 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933664+0000 mon.smithi097 (mon.0) 317 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933832+0000 mon.smithi097 (mon.0) 318 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.933993+0000 mon.smithi097 (mon.0) 319 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:35 smithi193 bash[28559]: audit 2023-11-17T06:50:34.934186+0000 mon.smithi097 (mon.0) 320 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:36 smithi193 bash[28559]: cluster 2023-11-17T06:50:35.312168+0000 mgr.smithi097.swvonl (mgr.14180) 92 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.766 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:36 smithi193 bash[28559]: audit 2023-11-17T06:50:36.350578+0000 mon.smithi193 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.193:0/1750204943' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.767 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:37 smithi193 bash[28559]: audit 2023-11-17T06:50:36.800113+0000 mon.smithi097 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.97:0/2145551594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.767 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:38 smithi193 bash[28559]: cluster 2023-11-17T06:50:37.312576+0000 mgr.smithi097.swvonl (mgr.14180) 93 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: audit 2023-11-17T06:50:02.996358+0000 mon.smithi097 (mon.0) 227 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: cephadm 2023-11-17T06:50:02.997334+0000 mgr.smithi097.swvonl (mgr.14180) 66 : cephadm [INF] Reconfiguring mgr.smithi193.zqrhit (monmap changed)... 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: audit 2023-11-17T06:50:02.997734+0000 mon.smithi097 (mon.0) 228 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi193.zqrhit", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: audit 2023-11-17T06:50:02.998846+0000 mon.smithi097 (mon.0) 229 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: audit 2023-11-17T06:50:02.999651+0000 mon.smithi097 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: cephadm 2023-11-17T06:50:03.000584+0000 mgr.smithi097.swvonl (mgr.14180) 67 : cephadm [INF] Reconfiguring daemon mgr.smithi193.zqrhit on smithi193 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:03 smithi097 bash[24958]: cluster 2023-11-17T06:50:03.304971+0000 mgr.smithi097.swvonl (mgr.14180) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: audit 2023-11-17T06:50:04.730222+0000 mon.smithi097 (mon.0) 231 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: cephadm 2023-11-17T06:50:04.731430+0000 mgr.smithi097.swvonl (mgr.14180) 69 : cephadm [INF] Reconfiguring mon.smithi193 (monmap changed)... 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: audit 2023-11-17T06:50:04.731963+0000 mon.smithi097 (mon.0) 232 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: audit 2023-11-17T06:50:04.733437+0000 mon.smithi097 (mon.0) 233 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-17T06:50:39.768 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: audit 2023-11-17T06:50:04.734944+0000 mon.smithi097 (mon.0) 234 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: cephadm 2023-11-17T06:50:04.736235+0000 mgr.smithi097.swvonl (mgr.14180) 70 : cephadm [INF] Reconfiguring daemon mon.smithi193 on smithi193 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:05 smithi097 bash[24958]: cluster 2023-11-17T06:50:05.305501+0000 mgr.smithi097.swvonl (mgr.14180) 71 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.436258+0000 mon.smithi097 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.441770+0000 mon.smithi097 (mon.0) 236 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.442601+0000 mgr.smithi097.swvonl (mgr.14180) 72 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.444628+0000 mon.smithi097 (mon.0) 237 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.445253+0000 mgr.smithi097.swvonl (mgr.14180) 73 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.451786+0000 mon.smithi097 (mon.0) 238 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.460696+0000 mon.smithi097 (mon.0) 239 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.461267+0000 mgr.smithi097.swvonl (mgr.14180) 74 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.466558+0000 mon.smithi097 (mon.0) 240 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.467092+0000 mgr.smithi097.swvonl (mgr.14180) 75 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.475668+0000 mon.smithi097 (mon.0) 241 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.481992+0000 mon.smithi097 (mon.0) 242 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.482801+0000 mgr.smithi097.swvonl (mgr.14180) 76 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.488730+0000 mon.smithi097 (mon.0) 243 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.489535+0000 mgr.smithi097.swvonl (mgr.14180) 77 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.496632+0000 mon.smithi097 (mon.0) 244 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:07 smithi097 bash[24958]: audit 2023-11-17T06:50:06.500631+0000 mon.smithi097 (mon.0) 245 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:08 smithi097 bash[24958]: cluster 2023-11-17T06:50:07.305996+0000 mgr.smithi097.swvonl (mgr.14180) 78 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:10 smithi097 bash[24958]: cluster 2023-11-17T06:50:09.306444+0000 mgr.smithi097.swvonl (mgr.14180) 79 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:11 smithi097 bash[24958]: cluster 2023-11-17T06:50:11.306837+0000 mgr.smithi097.swvonl (mgr.14180) 80 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:11 smithi097 bash[24958]: audit 2023-11-17T06:50:11.834567+0000 mon.smithi097 (mon.0) 246 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.143517+0000 mon.smithi097 (mon.0) 247 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.462485+0000 mon.smithi097 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.739697+0000 mon.smithi097 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.740627+0000 mon.smithi097 (mon.0) 250 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.741601+0000 mon.smithi097 (mon.0) 251 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.748405+0000 mon.smithi097 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.753903+0000 mon.smithi097 (mon.0) 253 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.760088+0000 mon.smithi097 (mon.0) 254 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.763492+0000 mon.smithi097 (mon.0) 255 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.763906+0000 mon.smithi097 (mon.0) 256 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:13 smithi097 bash[24958]: audit 2023-11-17T06:50:12.764807+0000 mon.smithi097 (mon.0) 257 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:14 smithi097 bash[24958]: cluster 2023-11-17T06:50:13.307291+0000 mgr.smithi097.swvonl (mgr.14180) 81 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:16 smithi097 bash[24958]: cluster 2023-11-17T06:50:15.307752+0000 mgr.smithi097.swvonl (mgr.14180) 82 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:18 smithi097 bash[24958]: cluster 2023-11-17T06:50:17.308175+0000 mgr.smithi097.swvonl (mgr.14180) 83 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:20 smithi097 bash[24958]: cluster 2023-11-17T06:50:19.308622+0000 mgr.smithi097.swvonl (mgr.14180) 84 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: cluster 2023-11-17T06:50:21.309082+0000 mgr.smithi097.swvonl (mgr.14180) 85 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.375232+0000 mon.smithi193 (mon.1) 2 : audit [INF] from='client.? 172.21.15.193:0/1809673918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]: dispatch 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.376511+0000 mon.smithi097 (mon.0) 258 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]: dispatch 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.383925+0000 mon.smithi097 (mon.0) 259 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e0cefb6-3c13-4136-bee5-9a4a3589ccd1"}]': finished 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: cluster 2023-11-17T06:50:21.384075+0000 mon.smithi097 (mon.0) 260 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.384359+0000 mon.smithi097 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.902120+0000 mon.smithi097 (mon.0) 262 : audit [INF] from='client.? 172.21.15.97:0/3308878968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe"}]: dispatch 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.909429+0000 mon.smithi097 (mon.0) 263 : audit [INF] from='client.? 172.21.15.97:0/3308878968' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe"}]': finished 2023-11-17T06:50:39.808 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: cluster 2023-11-17T06:50:21.909523+0000 mon.smithi097 (mon.0) 264 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.909756+0000 mon.smithi097 (mon.0) 265 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:21 smithi097 bash[24958]: audit 2023-11-17T06:50:21.910523+0000 mon.smithi097 (mon.0) 266 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:23 smithi097 bash[24958]: audit 2023-11-17T06:50:22.962079+0000 mon.smithi193 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.193:0/2451937437' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:24 smithi097 bash[24958]: cluster 2023-11-17T06:50:23.309438+0000 mgr.smithi097.swvonl (mgr.14180) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:24 smithi097 bash[24958]: audit 2023-11-17T06:50:23.464213+0000 mon.smithi097 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.97:0/1544928080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: cluster 2023-11-17T06:50:25.309870+0000 mgr.smithi097.swvonl (mgr.14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.759223+0000 mon.smithi193 (mon.1) 4 : audit [INF] from='client.? 172.21.15.193:0/2642303979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]: dispatch 2023-11-17T06:50:39.809 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.760344+0000 mon.smithi097 (mon.0) 268 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]: dispatch 2023-11-17T06:50:39.810 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.770166+0000 mon.smithi097 (mon.0) 269 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76c1caa8-14d7-43e5-bedf-44135cb7b231"}]': finished 2023-11-17T06:50:39.810 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: cluster 2023-11-17T06:50:25.770311+0000 mon.smithi097 (mon.0) 270 : cluster [DBG] osdmap e8: 3 total, 0 up, 3 in 2023-11-17T06:50:39.810 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.770504+0000 mon.smithi097 (mon.0) 271 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.810 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.771125+0000 mon.smithi097 (mon.0) 272 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.810 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:25.771451+0000 mon.smithi097 (mon.0) 273 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.253579+0000 mon.smithi097 (mon.0) 274 : audit [INF] from='client.? 172.21.15.97:0/3420491407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9bb2d027-be64-4f25-92eb-0d72ee3f968c"}]: dispatch 2023-11-17T06:50:39.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.259593+0000 mon.smithi097 (mon.0) 275 : audit [INF] from='client.? 172.21.15.97:0/3420491407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9bb2d027-be64-4f25-92eb-0d72ee3f968c"}]': finished 2023-11-17T06:50:39.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: cluster 2023-11-17T06:50:26.259642+0000 mon.smithi097 (mon.0) 276 : cluster [DBG] osdmap e9: 4 total, 0 up, 4 in 2023-11-17T06:50:39.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.259727+0000 mon.smithi097 (mon.0) 277 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.811 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.259969+0000 mon.smithi097 (mon.0) 278 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.260128+0000 mon.smithi097 (mon.0) 279 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:26 smithi097 bash[24958]: audit 2023-11-17T06:50:26.260378+0000 mon.smithi097 (mon.0) 280 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:27 smithi097 bash[24958]: audit 2023-11-17T06:50:27.314193+0000 mon.smithi193 (mon.1) 5 : audit [DBG] from='client.? 172.21.15.193:0/2197922542' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:28 smithi097 bash[24958]: cluster 2023-11-17T06:50:27.310369+0000 mgr.smithi097.swvonl (mgr.14180) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:28 smithi097 bash[24958]: audit 2023-11-17T06:50:27.792075+0000 mon.smithi097 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.97:0/2838779727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: cluster 2023-11-17T06:50:29.310817+0000 mgr.smithi097.swvonl (mgr.14180) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.812 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.108471+0000 mon.smithi193 (mon.1) 6 : audit [INF] from='client.? 172.21.15.193:0/882179693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]: dispatch 2023-11-17T06:50:39.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.109629+0000 mon.smithi097 (mon.0) 282 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]: dispatch 2023-11-17T06:50:39.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.120211+0000 mon.smithi097 (mon.0) 283 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3"}]': finished 2023-11-17T06:50:39.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: cluster 2023-11-17T06:50:30.120335+0000 mon.smithi097 (mon.0) 284 : cluster [DBG] osdmap e10: 5 total, 0 up, 5 in 2023-11-17T06:50:39.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.120612+0000 mon.smithi097 (mon.0) 285 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.813 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.121256+0000 mon.smithi097 (mon.0) 286 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.121658+0000 mon.smithi097 (mon.0) 287 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.122060+0000 mon.smithi097 (mon.0) 288 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:30 smithi097 bash[24958]: audit 2023-11-17T06:50:30.122471+0000 mon.smithi097 (mon.0) 289 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.575312+0000 mon.smithi097 (mon.0) 290 : audit [INF] from='client.? 172.21.15.97:0/270524419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7463f212-d4f5-427f-a4db-3835fce07e46"}]: dispatch 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.580805+0000 mon.smithi097 (mon.0) 291 : audit [INF] from='client.? 172.21.15.97:0/270524419' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7463f212-d4f5-427f-a4db-3835fce07e46"}]': finished 2023-11-17T06:50:39.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: cluster 2023-11-17T06:50:30.580843+0000 mon.smithi097 (mon.0) 292 : cluster [DBG] osdmap e11: 6 total, 0 up, 6 in 2023-11-17T06:50:39.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.581002+0000 mon.smithi097 (mon.0) 293 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.581605+0000 mon.smithi097 (mon.0) 294 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.582065+0000 mon.smithi097 (mon.0) 295 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.582552+0000 mon.smithi097 (mon.0) 296 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.583027+0000 mon.smithi097 (mon.0) 297 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.816 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:31 smithi097 bash[24958]: audit 2023-11-17T06:50:30.583435+0000 mon.smithi097 (mon.0) 298 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.816 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:32 smithi097 bash[24958]: cluster 2023-11-17T06:50:31.311194+0000 mgr.smithi097.swvonl (mgr.14180) 90 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.816 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:32 smithi097 bash[24958]: audit 2023-11-17T06:50:31.729490+0000 mon.smithi193 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.193:0/3468112770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.816 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:32 smithi097 bash[24958]: audit 2023-11-17T06:50:32.117927+0000 mon.smithi097 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.97:0/2965368618' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:34 smithi097 bash[24958]: cluster 2023-11-17T06:50:33.311680+0000 mgr.smithi097.swvonl (mgr.14180) 91 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.538818+0000 mon.smithi193 (mon.1) 8 : audit [INF] from='client.? 172.21.15.193:0/3242142143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]: dispatch 2023-11-17T06:50:39.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.540014+0000 mon.smithi097 (mon.0) 300 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]: dispatch 2023-11-17T06:50:39.817 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.546732+0000 mon.smithi097 (mon.0) 301 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e7230c8-6a76-4ced-a43f-63071d3dde82"}]': finished 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: cluster 2023-11-17T06:50:34.546847+0000 mon.smithi097 (mon.0) 302 : cluster [DBG] osdmap e12: 7 total, 0 up, 7 in 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.547051+0000 mon.smithi097 (mon.0) 303 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.547817+0000 mon.smithi097 (mon.0) 304 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.548232+0000 mon.smithi097 (mon.0) 305 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.548599+0000 mon.smithi097 (mon.0) 306 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.548965+0000 mon.smithi097 (mon.0) 307 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.818 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.549326+0000 mon.smithi097 (mon.0) 308 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.549688+0000 mon.smithi097 (mon.0) 309 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.924378+0000 mon.smithi097 (mon.0) 310 : audit [INF] from='client.? 172.21.15.97:0/226756965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4c1b9cb-3792-496d-b445-63a1e16c3823"}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.932510+0000 mon.smithi097 (mon.0) 311 : audit [INF] from='client.? 172.21.15.97:0/226756965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4c1b9cb-3792-496d-b445-63a1e16c3823"}]': finished 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: cluster 2023-11-17T06:50:34.932674+0000 mon.smithi097 (mon.0) 312 : cluster [DBG] osdmap e13: 8 total, 0 up, 8 in 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.932835+0000 mon.smithi097 (mon.0) 313 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933089+0000 mon.smithi097 (mon.0) 314 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933293+0000 mon.smithi097 (mon.0) 315 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933458+0000 mon.smithi097 (mon.0) 316 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933664+0000 mon.smithi097 (mon.0) 317 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933832+0000 mon.smithi097 (mon.0) 318 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.933993+0000 mon.smithi097 (mon.0) 319 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:35 smithi097 bash[24958]: audit 2023-11-17T06:50:34.934186+0000 mon.smithi097 (mon.0) 320 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:36 smithi097 bash[24958]: cluster 2023-11-17T06:50:35.312168+0000 mgr.smithi097.swvonl (mgr.14180) 92 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:39.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:36 smithi097 bash[24958]: audit 2023-11-17T06:50:36.350578+0000 mon.smithi193 (mon.1) 9 : audit [DBG] from='client.? 172.21.15.193:0/1750204943' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:37 smithi097 bash[24958]: audit 2023-11-17T06:50:36.800113+0000 mon.smithi097 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.97:0/2145551594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-17T06:50:39.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:38 smithi097 bash[24958]: cluster 2023-11-17T06:50:37.312576+0000 mgr.smithi097.swvonl (mgr.14180) 93 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:40.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:50:40.726 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:40 smithi097 bash[24958]: cluster 2023-11-17T06:50:39.312999+0000 mgr.smithi097.swvonl (mgr.14180) 94 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:40.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:40 smithi193 bash[28559]: cluster 2023-11-17T06:50:39.312999+0000 mgr.smithi097.swvonl (mgr.14180) 94 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:42.173 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:41 smithi097 bash[24958]: cluster 2023-11-17T06:50:41.313458+0000 mgr.smithi097.swvonl (mgr.14180) 95 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:42.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:41 smithi193 bash[28559]: cluster 2023-11-17T06:50:41.313458+0000 mgr.smithi097.swvonl (mgr.14180) 95 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:43.939 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:50:44.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:44 smithi097 bash[24958]: cluster 2023-11-17T06:50:43.313969+0000 mgr.smithi097.swvonl (mgr.14180) 96 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:44.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:44 smithi097 bash[24958]: audit 2023-11-17T06:50:43.937924+0000 mon.smithi097 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.97:0/2394083575' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:44.729 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:50:44.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:44 smithi193 bash[28559]: cluster 2023-11-17T06:50:43.313969+0000 mgr.smithi097.swvonl (mgr.14180) 96 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:44.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:44 smithi193 bash[28559]: audit 2023-11-17T06:50:43.937924+0000 mon.smithi097 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.97:0/2394083575' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:45.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:50:46.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:46 smithi193 bash[28559]: cluster 2023-11-17T06:50:45.314466+0000 mgr.smithi097.swvonl (mgr.14180) 97 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:46.766 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:46 smithi097 bash[24958]: cluster 2023-11-17T06:50:45.314466+0000 mgr.smithi097.swvonl (mgr.14180) 97 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:48.797 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:48 smithi097 bash[24958]: cluster 2023-11-17T06:50:47.314783+0000 mgr.smithi097.swvonl (mgr.14180) 98 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:48.797 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:48 smithi097 bash[24958]: audit 2023-11-17T06:50:48.213194+0000 mon.smithi097 (mon.0) 323 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-17T06:50:48.797 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:48 smithi097 bash[24958]: audit 2023-11-17T06:50:48.214482+0000 mon.smithi097 (mon.0) 324 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:48.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:48 smithi193 bash[28559]: cluster 2023-11-17T06:50:47.314783+0000 mgr.smithi097.swvonl (mgr.14180) 98 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:48.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:48 smithi193 bash[28559]: audit 2023-11-17T06:50:48.213194+0000 mon.smithi097 (mon.0) 323 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-17T06:50:48.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:48 smithi193 bash[28559]: audit 2023-11-17T06:50:48.214482+0000 mon.smithi097 (mon.0) 324 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:49.529 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:49 smithi193 bash[28559]: cephadm 2023-11-17T06:50:48.215677+0000 mgr.smithi097.swvonl (mgr.14180) 99 : cephadm [INF] Deploying daemon osd.0 on smithi193 2023-11-17T06:50:51.036 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:49 smithi097 bash[24958]: cephadm 2023-11-17T06:50:48.215677+0000 mgr.smithi097.swvonl (mgr.14180) 99 : cephadm [INF] Deploying daemon osd.0 on smithi193 2023-11-17T06:50:51.037 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:50 smithi193 bash[28559]: cluster 2023-11-17T06:50:49.315094+0000 mgr.smithi097.swvonl (mgr.14180) 100 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:51.037 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:50:51.037 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:50 smithi097 bash[24958]: cluster 2023-11-17T06:50:49.315094+0000 mgr.smithi097.swvonl (mgr.14180) 100 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:51.512 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:51 smithi097 bash[24958]: audit 2023-11-17T06:50:50.424931+0000 mon.smithi097 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.97:0/1914151243' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:51.513 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:50:51.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:51 smithi193 bash[28559]: audit 2023-11-17T06:50:50.424931+0000 mon.smithi097 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.97:0/1914151243' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:52.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:50:52.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: cluster 2023-11-17T06:50:51.315598+0000 mgr.smithi097.swvonl (mgr.14180) 101 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: audit 2023-11-17T06:50:51.450649+0000 mon.smithi097 (mon.0) 326 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: audit 2023-11-17T06:50:51.452064+0000 mon.smithi097 (mon.0) 327 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: cephadm 2023-11-17T06:50:51.453050+0000 mgr.smithi097.swvonl (mgr.14180) 102 : cephadm [INF] Deploying daemon osd.1 on smithi097 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: audit 2023-11-17T06:50:52.226027+0000 mon.smithi097 (mon.0) 328 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: audit 2023-11-17T06:50:52.227353+0000 mon.smithi097 (mon.0) 329 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-17T06:50:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:52 smithi193 bash[28559]: audit 2023-11-17T06:50:52.228944+0000 mon.smithi097 (mon.0) 330 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: cluster 2023-11-17T06:50:51.315598+0000 mgr.smithi097.swvonl (mgr.14180) 101 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: audit 2023-11-17T06:50:51.450649+0000 mon.smithi097 (mon.0) 326 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: audit 2023-11-17T06:50:51.452064+0000 mon.smithi097 (mon.0) 327 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: cephadm 2023-11-17T06:50:51.453050+0000 mgr.smithi097.swvonl (mgr.14180) 102 : cephadm [INF] Deploying daemon osd.1 on smithi097 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: audit 2023-11-17T06:50:52.226027+0000 mon.smithi097 (mon.0) 328 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: audit 2023-11-17T06:50:52.227353+0000 mon.smithi097 (mon.0) 329 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-17T06:50:52.647 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:52 smithi097 bash[24958]: audit 2023-11-17T06:50:52.228944+0000 mon.smithi097 (mon.0) 330 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:53.747 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:53 smithi097 bash[24958]: cephadm 2023-11-17T06:50:52.230209+0000 mgr.smithi097.swvonl (mgr.14180) 103 : cephadm [INF] Deploying daemon osd.2 on smithi193 2023-11-17T06:50:53.773 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:53 smithi193 bash[28559]: cephadm 2023-11-17T06:50:52.230209+0000 mgr.smithi097.swvonl (mgr.14180) 103 : cephadm [INF] Deploying daemon osd.2 on smithi193 2023-11-17T06:50:54.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:54 smithi097 bash[24958]: cluster 2023-11-17T06:50:53.315910+0000 mgr.smithi097.swvonl (mgr.14180) 104 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:54.905 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:54 smithi193 bash[28559]: cluster 2023-11-17T06:50:53.315910+0000 mgr.smithi097.swvonl (mgr.14180) 104 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:56.650 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:56 smithi193 bash[28559]: cluster 2023-11-17T06:50:55.316256+0000 mgr.smithi097.swvonl (mgr.14180) 105 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:56.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:56 smithi097 bash[24958]: cluster 2023-11-17T06:50:55.316256+0000 mgr.smithi097.swvonl (mgr.14180) 105 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:58.171 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:50:58.715 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: cluster 2023-11-17T06:50:57.316533+0000 mgr.smithi097.swvonl (mgr.14180) 106 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:57.534995+0000 mon.smithi097 (mon.0) 331 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:57.536058+0000 mon.smithi097 (mon.0) 332 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:57.537321+0000 mon.smithi097 (mon.0) 333 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: cephadm 2023-11-17T06:50:57.538555+0000 mgr.smithi097.swvonl (mgr.14180) 107 : cephadm [INF] Deploying daemon osd.4 on smithi193 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:58.169121+0000 mon.smithi193 (mon.1) 10 : audit [DBG] from='client.? 172.21.15.97:0/2009870783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:58.301021+0000 mon.smithi097 (mon.0) 334 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:58.302063+0000 mon.smithi097 (mon.0) 335 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-17T06:50:58.716 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:58 smithi097 bash[24958]: audit 2023-11-17T06:50:58.303012+0000 mon.smithi097 (mon.0) 336 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:58.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: cluster 2023-11-17T06:50:57.316533+0000 mgr.smithi097.swvonl (mgr.14180) 106 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:57.534995+0000 mon.smithi097 (mon.0) 331 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:57.536058+0000 mon.smithi097 (mon.0) 332 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:57.537321+0000 mon.smithi097 (mon.0) 333 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: cephadm 2023-11-17T06:50:57.538555+0000 mgr.smithi097.swvonl (mgr.14180) 107 : cephadm [INF] Deploying daemon osd.4 on smithi193 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:58.169121+0000 mon.smithi193 (mon.1) 10 : audit [DBG] from='client.? 172.21.15.97:0/2009870783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:58.301021+0000 mon.smithi097 (mon.0) 334 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:58.302063+0000 mon.smithi097 (mon.0) 335 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-17T06:50:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:58 smithi193 bash[28559]: audit 2023-11-17T06:50:58.303012+0000 mon.smithi097 (mon.0) 336 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:50:59.552 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":13,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:50:59.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:50:59 smithi097 bash[24958]: cephadm 2023-11-17T06:50:58.303761+0000 mgr.smithi097.swvonl (mgr.14180) 108 : cephadm [INF] Deploying daemon osd.3 on smithi097 2023-11-17T06:50:59.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:50:59 smithi193 bash[28559]: cephadm 2023-11-17T06:50:58.303761+0000 mgr.smithi097.swvonl (mgr.14180) 108 : cephadm [INF] Deploying daemon osd.3 on smithi097 2023-11-17T06:51:00.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:51:00.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:00 smithi193 bash[28559]: cluster 2023-11-17T06:50:59.316899+0000 mgr.smithi097.swvonl (mgr.14180) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:00.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:00 smithi193 bash[28559]: audit 2023-11-17T06:51:00.304687+0000 mon.smithi193 (mon.1) 11 : audit [INF] from='osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T06:51:00.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:00 smithi193 bash[28559]: audit 2023-11-17T06:51:00.305602+0000 mon.smithi097 (mon.0) 337 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T06:51:00.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:00 smithi097 bash[24958]: cluster 2023-11-17T06:50:59.316899+0000 mgr.smithi097.swvonl (mgr.14180) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:00.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:00 smithi097 bash[24958]: audit 2023-11-17T06:51:00.304687+0000 mon.smithi193 (mon.1) 11 : audit [INF] from='osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T06:51:00.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:00 smithi097 bash[24958]: audit 2023-11-17T06:51:00.305602+0000 mon.smithi097 (mon.0) 337 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.553262+0000 mon.smithi097 (mon.0) 338 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: cluster 2023-11-17T06:51:00.553353+0000 mon.smithi097 (mon.0) 339 : cluster [DBG] osdmap e14: 8 total, 0 up, 8 in 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.553513+0000 mon.smithi097 (mon.0) 340 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.553915+0000 mon.smithi097 (mon.0) 341 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554159+0000 mon.smithi097 (mon.0) 342 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554380+0000 mon.smithi193 (mon.1) 12 : audit [INF] from='osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554434+0000 mon.smithi097 (mon.0) 343 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:01.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554623+0000 mon.smithi097 (mon.0) 344 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:01.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554795+0000 mon.smithi097 (mon.0) 345 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:01.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.554967+0000 mon.smithi097 (mon.0) 346 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:01.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.555231+0000 mon.smithi097 (mon.0) 347 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:01.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:01 smithi193 bash[28559]: audit 2023-11-17T06:51:00.555534+0000 mon.smithi097 (mon.0) 348 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:02.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.553262+0000 mon.smithi097 (mon.0) 338 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: cluster 2023-11-17T06:51:00.553353+0000 mon.smithi097 (mon.0) 339 : cluster [DBG] osdmap e14: 8 total, 0 up, 8 in 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.553513+0000 mon.smithi097 (mon.0) 340 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.553915+0000 mon.smithi097 (mon.0) 341 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554159+0000 mon.smithi097 (mon.0) 342 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554380+0000 mon.smithi193 (mon.1) 12 : audit [INF] from='osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554434+0000 mon.smithi097 (mon.0) 343 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554623+0000 mon.smithi097 (mon.0) 344 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554795+0000 mon.smithi097 (mon.0) 345 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.554967+0000 mon.smithi097 (mon.0) 346 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.555231+0000 mon.smithi097 (mon.0) 347 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:01 smithi097 bash[24958]: audit 2023-11-17T06:51:00.555534+0000 mon.smithi097 (mon.0) 348 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: cluster 2023-11-17T06:51:01.317138+0000 mgr.smithi097.swvonl (mgr.14180) 110 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.556371+0000 mon.smithi097 (mon.0) 349 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: cluster 2023-11-17T06:51:01.556465+0000 mon.smithi097 (mon.0) 350 : cluster [DBG] osdmap e15: 8 total, 0 up, 8 in 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.556961+0000 mon.smithi097 (mon.0) 351 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.557305+0000 mon.smithi097 (mon.0) 352 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.557491+0000 mon.smithi097 (mon.0) 353 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.557668+0000 mon.smithi097 (mon.0) 354 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:02.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.557956+0000 mon.smithi097 (mon.0) 355 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:02.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.558159+0000 mon.smithi097 (mon.0) 356 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:02.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.558349+0000 mon.smithi097 (mon.0) 357 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:02.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.558528+0000 mon.smithi097 (mon.0) 358 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:02.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:02 smithi193 bash[28559]: audit 2023-11-17T06:51:01.564085+0000 mon.smithi097 (mon.0) 359 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:02.917 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: cluster 2023-11-17T06:51:01.317138+0000 mgr.smithi097.swvonl (mgr.14180) 110 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:02.917 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.556371+0000 mon.smithi097 (mon.0) 349 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: cluster 2023-11-17T06:51:01.556465+0000 mon.smithi097 (mon.0) 350 : cluster [DBG] osdmap e15: 8 total, 0 up, 8 in 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.556961+0000 mon.smithi097 (mon.0) 351 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.557305+0000 mon.smithi097 (mon.0) 352 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.557491+0000 mon.smithi097 (mon.0) 353 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.557668+0000 mon.smithi097 (mon.0) 354 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.557956+0000 mon.smithi097 (mon.0) 355 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.558159+0000 mon.smithi097 (mon.0) 356 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.558349+0000 mon.smithi097 (mon.0) 357 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.558528+0000 mon.smithi097 (mon.0) 358 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:02.918 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:02 smithi097 bash[24958]: audit 2023-11-17T06:51:01.564085+0000 mon.smithi097 (mon.0) 359 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:03.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: cluster 2023-11-17T06:51:01.239614+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: cluster 2023-11-17T06:51:01.239708+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: cluster 2023-11-17T06:51:02.564892+0000 mon.smithi097 (mon.0) 360 : cluster [INF] osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844] boot 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: cluster 2023-11-17T06:51:02.564959+0000 mon.smithi097 (mon.0) 361 : cluster [DBG] osdmap e16: 8 total, 1 up, 8 in 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.565367+0000 mon.smithi097 (mon.0) 362 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.565659+0000 mon.smithi097 (mon.0) 363 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.565848+0000 mon.smithi097 (mon.0) 364 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.566115+0000 mon.smithi097 (mon.0) 365 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.566304+0000 mon.smithi097 (mon.0) 366 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.569817+0000 mon.smithi097 (mon.0) 367 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.570015+0000 mon.smithi097 (mon.0) 368 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:02.570222+0000 mon.smithi097 (mon.0) 369 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:03.418183+0000 mon.smithi097 (mon.0) 370 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:03.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:03.419609+0000 mon.smithi097 (mon.0) 371 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-17T06:51:03.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:03 smithi193 bash[28559]: audit 2023-11-17T06:51:03.420876+0000 mon.smithi097 (mon.0) 372 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:04.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: cluster 2023-11-17T06:51:01.239614+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: cluster 2023-11-17T06:51:01.239708+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: cluster 2023-11-17T06:51:02.564892+0000 mon.smithi097 (mon.0) 360 : cluster [INF] osd.0 [v2:172.21.15.193:6800/2497341844,v1:172.21.15.193:6801/2497341844] boot 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: cluster 2023-11-17T06:51:02.564959+0000 mon.smithi097 (mon.0) 361 : cluster [DBG] osdmap e16: 8 total, 1 up, 8 in 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.565367+0000 mon.smithi097 (mon.0) 362 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.565659+0000 mon.smithi097 (mon.0) 363 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.565848+0000 mon.smithi097 (mon.0) 364 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.566115+0000 mon.smithi097 (mon.0) 365 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.566304+0000 mon.smithi097 (mon.0) 366 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.569817+0000 mon.smithi097 (mon.0) 367 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:04.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.570015+0000 mon.smithi097 (mon.0) 368 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:04.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:02.570222+0000 mon.smithi097 (mon.0) 369 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:04.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:03.418183+0000 mon.smithi097 (mon.0) 370 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:04.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:03.419609+0000 mon.smithi097 (mon.0) 371 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-17T06:51:04.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:03 smithi097 bash[24958]: audit 2023-11-17T06:51:03.420876+0000 mon.smithi097 (mon.0) 372 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: cluster 2023-11-17T06:51:03.317458+0000 mgr.smithi097.swvonl (mgr.14180) 111 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: cephadm 2023-11-17T06:51:03.421683+0000 mgr.smithi097.swvonl (mgr.14180) 112 : cephadm [INF] Deploying daemon osd.6 on smithi193 2023-11-17T06:51:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: cluster 2023-11-17T06:51:03.577158+0000 mon.smithi097 (mon.0) 373 : cluster [DBG] osdmap e17: 8 total, 1 up, 8 in 2023-11-17T06:51:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.577286+0000 mon.smithi097 (mon.0) 374 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.577661+0000 mon.smithi097 (mon.0) 375 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.577816+0000 mon.smithi097 (mon.0) 376 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.577987+0000 mon.smithi097 (mon.0) 377 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.578204+0000 mon.smithi097 (mon.0) 378 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.578424+0000 mon.smithi097 (mon.0) 379 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:04 smithi097 bash[24958]: audit 2023-11-17T06:51:03.578781+0000 mon.smithi097 (mon.0) 380 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: cluster 2023-11-17T06:51:03.317458+0000 mgr.smithi097.swvonl (mgr.14180) 111 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: cephadm 2023-11-17T06:51:03.421683+0000 mgr.smithi097.swvonl (mgr.14180) 112 : cephadm [INF] Deploying daemon osd.6 on smithi193 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: cluster 2023-11-17T06:51:03.577158+0000 mon.smithi097 (mon.0) 373 : cluster [DBG] osdmap e17: 8 total, 1 up, 8 in 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.577286+0000 mon.smithi097 (mon.0) 374 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.577661+0000 mon.smithi097 (mon.0) 375 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.577816+0000 mon.smithi097 (mon.0) 376 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.577987+0000 mon.smithi097 (mon.0) 377 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.578204+0000 mon.smithi097 (mon.0) 378 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.578424+0000 mon.smithi097 (mon.0) 379 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:04 smithi193 bash[28559]: audit 2023-11-17T06:51:03.578781+0000 mon.smithi097 (mon.0) 380 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:06.648 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:06.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: cluster 2023-11-17T06:51:05.317960+0000 mgr.smithi097.swvonl (mgr.14180) 113 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:06.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: audit 2023-11-17T06:51:05.878731+0000 mon.smithi193 (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T06:51:06.819 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: audit 2023-11-17T06:51:05.879605+0000 mon.smithi097 (mon.0) 381 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T06:51:06.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: audit 2023-11-17T06:51:06.442314+0000 mon.smithi097 (mon.0) 382 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:06.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: audit 2023-11-17T06:51:06.443361+0000 mon.smithi097 (mon.0) 383 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-17T06:51:06.820 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:06 smithi097 bash[24958]: audit 2023-11-17T06:51:06.444183+0000 mon.smithi097 (mon.0) 384 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:07.035 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: cluster 2023-11-17T06:51:05.317960+0000 mgr.smithi097.swvonl (mgr.14180) 113 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:07.035 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: audit 2023-11-17T06:51:05.878731+0000 mon.smithi193 (mon.1) 13 : audit [INF] from='osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T06:51:07.035 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: audit 2023-11-17T06:51:05.879605+0000 mon.smithi097 (mon.0) 381 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-17T06:51:07.035 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: audit 2023-11-17T06:51:06.442314+0000 mon.smithi097 (mon.0) 382 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:07.036 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: audit 2023-11-17T06:51:06.443361+0000 mon.smithi097 (mon.0) 383 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-17T06:51:07.036 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:06 smithi193 bash[28559]: audit 2023-11-17T06:51:06.444183+0000 mon.smithi097 (mon.0) 384 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:07.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: cephadm 2023-11-17T06:51:06.444852+0000 mgr.smithi097.swvonl (mgr.14180) 114 : cephadm [INF] Deploying daemon osd.5 on smithi097 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.589824+0000 mon.smithi097 (mon.0) 385 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: cluster 2023-11-17T06:51:06.589923+0000 mon.smithi097 (mon.0) 386 : cluster [DBG] osdmap e18: 8 total, 1 up, 8 in 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.590062+0000 mon.smithi097 (mon.0) 387 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.590406+0000 mon.smithi097 (mon.0) 388 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.590620+0000 mon.smithi097 (mon.0) 389 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.590698+0000 mon.smithi193 (mon.1) 14 : audit [INF] from='osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.590830+0000 mon.smithi097 (mon.0) 390 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.591045+0000 mon.smithi097 (mon.0) 391 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.591282+0000 mon.smithi097 (mon.0) 392 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:07.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.591738+0000 mon.smithi097 (mon.0) 393 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:07.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.592102+0000 mon.smithi097 (mon.0) 394 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:07.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:07 smithi097 bash[24958]: audit 2023-11-17T06:51:06.647010+0000 mon.smithi097 (mon.0) 395 : audit [DBG] from='client.? 172.21.15.97:0/2226283976' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: cephadm 2023-11-17T06:51:06.444852+0000 mgr.smithi097.swvonl (mgr.14180) 114 : cephadm [INF] Deploying daemon osd.5 on smithi097 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.589824+0000 mon.smithi097 (mon.0) 385 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: cluster 2023-11-17T06:51:06.589923+0000 mon.smithi097 (mon.0) 386 : cluster [DBG] osdmap e18: 8 total, 1 up, 8 in 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.590062+0000 mon.smithi097 (mon.0) 387 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.590406+0000 mon.smithi097 (mon.0) 388 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.590620+0000 mon.smithi097 (mon.0) 389 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.590698+0000 mon.smithi193 (mon.1) 14 : audit [INF] from='osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.590830+0000 mon.smithi097 (mon.0) 390 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.591045+0000 mon.smithi097 (mon.0) 391 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:07.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.591282+0000 mon.smithi097 (mon.0) 392 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:07.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.591738+0000 mon.smithi097 (mon.0) 393 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:07.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.592102+0000 mon.smithi097 (mon.0) 394 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:07.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:07 smithi193 bash[28559]: audit 2023-11-17T06:51:06.647010+0000 mon.smithi097 (mon.0) 395 : audit [DBG] from='client.? 172.21.15.97:0/2226283976' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:08.260 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":18,"num_osds":8,"num_up_osds":1,"osd_up_since":1700203862,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:51:08.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: cluster 2023-11-17T06:51:07.318503+0000 mgr.smithi097.swvonl (mgr.14180) 115 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:08.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.592844+0000 mon.smithi097 (mon.0) 396 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:08.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: cluster 2023-11-17T06:51:07.593004+0000 mon.smithi097 (mon.0) 397 : cluster [DBG] osdmap e19: 8 total, 1 up, 8 in 2023-11-17T06:51:08.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.593696+0000 mon.smithi097 (mon.0) 398 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:08.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.593975+0000 mon.smithi097 (mon.0) 399 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.594183+0000 mon.smithi097 (mon.0) 400 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.594419+0000 mon.smithi097 (mon.0) 401 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.594589+0000 mon.smithi097 (mon.0) 402 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.594769+0000 mon.smithi097 (mon.0) 403 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.594990+0000 mon.smithi097 (mon.0) 404 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:07.600796+0000 mon.smithi097 (mon.0) 405 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:08.719 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:08 smithi193 bash[28559]: audit 2023-11-17T06:51:08.392712+0000 mon.smithi097 (mon.0) 406 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-17T06:51:08.909 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: cluster 2023-11-17T06:51:07.318503+0000 mgr.smithi097.swvonl (mgr.14180) 115 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.592844+0000 mon.smithi097 (mon.0) 396 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: cluster 2023-11-17T06:51:07.593004+0000 mon.smithi097 (mon.0) 397 : cluster [DBG] osdmap e19: 8 total, 1 up, 8 in 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.593696+0000 mon.smithi097 (mon.0) 398 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.593975+0000 mon.smithi097 (mon.0) 399 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.594183+0000 mon.smithi097 (mon.0) 400 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.594419+0000 mon.smithi097 (mon.0) 401 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.594589+0000 mon.smithi097 (mon.0) 402 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.594769+0000 mon.smithi097 (mon.0) 403 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.594990+0000 mon.smithi097 (mon.0) 404 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:07.600796+0000 mon.smithi097 (mon.0) 405 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:08.910 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:08 smithi097 bash[24958]: audit 2023-11-17T06:51:08.392712+0000 mon.smithi097 (mon.0) 406 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-17T06:51:09.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:51:09.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: cluster 2023-11-17T06:51:06.883691+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:11.374 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: cluster 2023-11-17T06:51:06.883754+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:11.374 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: cluster 2023-11-17T06:51:08.602898+0000 mon.smithi097 (mon.0) 407 : cluster [INF] osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302] boot 2023-11-17T06:51:11.374 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.602996+0000 mon.smithi097 (mon.0) 408 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-17T06:51:11.374 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: cluster 2023-11-17T06:51:08.603122+0000 mon.smithi097 (mon.0) 409 : cluster [DBG] osdmap e20: 8 total, 2 up, 8 in 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.603693+0000 mon.smithi097 (mon.0) 410 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.604256+0000 mon.smithi097 (mon.0) 411 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.604665+0000 mon.smithi097 (mon.0) 412 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.610466+0000 mon.smithi097 (mon.0) 413 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.610800+0000 mon.smithi097 (mon.0) 414 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:11.375 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.611076+0000 mon.smithi097 (mon.0) 415 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:11.376 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.611454+0000 mon.smithi097 (mon.0) 416 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:11.376 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:08.611796+0000 mon.smithi097 (mon.0) 417 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:11.376 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:09.479508+0000 mon.smithi097 (mon.0) 418 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:11.376 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:09.603779+0000 mon.smithi097 (mon.0) 419 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:11.376 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: cluster 2023-11-17T06:51:09.603855+0000 mon.smithi097 (mon.0) 420 : cluster [DBG] osdmap e21: 8 total, 2 up, 8 in 2023-11-17T06:51:11.377 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:09.604143+0000 mon.smithi097 (mon.0) 421 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.377 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:09 smithi193 bash[28559]: audit 2023-11-17T06:51:09.604710+0000 mon.smithi097 (mon.0) 422 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:11.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: cluster 2023-11-17T06:51:06.883691+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:11.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: cluster 2023-11-17T06:51:06.883754+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:11.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: cluster 2023-11-17T06:51:08.602898+0000 mon.smithi097 (mon.0) 407 : cluster [INF] osd.2 [v2:172.21.15.193:6808/3825756302,v1:172.21.15.193:6809/3825756302] boot 2023-11-17T06:51:11.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.602996+0000 mon.smithi097 (mon.0) 408 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-17T06:51:11.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: cluster 2023-11-17T06:51:08.603122+0000 mon.smithi097 (mon.0) 409 : cluster [DBG] osdmap e20: 8 total, 2 up, 8 in 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.603693+0000 mon.smithi097 (mon.0) 410 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.604256+0000 mon.smithi097 (mon.0) 411 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.604665+0000 mon.smithi097 (mon.0) 412 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.610466+0000 mon.smithi097 (mon.0) 413 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.610800+0000 mon.smithi097 (mon.0) 414 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:11.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.611076+0000 mon.smithi097 (mon.0) 415 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.611454+0000 mon.smithi097 (mon.0) 416 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:08.611796+0000 mon.smithi097 (mon.0) 417 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:09.479508+0000 mon.smithi097 (mon.0) 418 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:09.603779+0000 mon.smithi097 (mon.0) 419 : audit [INF] from='osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: cluster 2023-11-17T06:51:09.603855+0000 mon.smithi097 (mon.0) 420 : cluster [DBG] osdmap e21: 8 total, 2 up, 8 in 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:09.604143+0000 mon.smithi097 (mon.0) 421 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.380 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:09 smithi097 bash[24958]: audit 2023-11-17T06:51:09.604710+0000 mon.smithi097 (mon.0) 422 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: cluster 2023-11-17T06:51:09.318866+0000 mgr.smithi097.swvonl (mgr.14180) 116 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:09.606004+0000 mon.smithi097 (mon.0) 423 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:09.606687+0000 mon.smithi097 (mon.0) 424 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:09.607147+0000 mon.smithi097 (mon.0) 425 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:09.607601+0000 mon.smithi097 (mon.0) 426 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:09.611240+0000 mon.smithi097 (mon.0) 427 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.437 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:11 smithi193 bash[28559]: audit 2023-11-17T06:51:10.610706+0000 mon.smithi097 (mon.0) 428 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: cluster 2023-11-17T06:51:09.318866+0000 mgr.smithi097.swvonl (mgr.14180) 116 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:09.606004+0000 mon.smithi097 (mon.0) 423 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:09.606687+0000 mon.smithi097 (mon.0) 424 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:09.607147+0000 mon.smithi097 (mon.0) 425 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:09.607601+0000 mon.smithi097 (mon.0) 426 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:09.611240+0000 mon.smithi097 (mon.0) 427 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:11 smithi097 bash[24958]: audit 2023-11-17T06:51:10.610706+0000 mon.smithi097 (mon.0) 428 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:09.387210+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:09.387271+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:10.616457+0000 mon.smithi097 (mon.0) 429 : cluster [INF] osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368] boot 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:10.616487+0000 mon.smithi097 (mon.0) 430 : cluster [DBG] osdmap e22: 8 total, 3 up, 8 in 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.616979+0000 mon.smithi097 (mon.0) 431 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.617248+0000 mon.smithi097 (mon.0) 432 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.617447+0000 mon.smithi097 (mon.0) 433 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.617659+0000 mon.smithi097 (mon.0) 434 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.617826+0000 mon.smithi097 (mon.0) 435 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.618025+0000 mon.smithi097 (mon.0) 436 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:10.618270+0000 mon.smithi097 (mon.0) 437 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-11-17T06:51:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:11.319219+0000 mgr.smithi097.swvonl (mgr.14180) 117 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.618500+0000 mon.smithi097 (mon.0) 438 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: cluster 2023-11-17T06:51:11.618569+0000 mon.smithi097 (mon.0) 439 : cluster [DBG] osdmap e23: 8 total, 3 up, 8 in 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.618835+0000 mon.smithi097 (mon.0) 440 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.619684+0000 mon.smithi097 (mon.0) 441 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.620211+0000 mon.smithi097 (mon.0) 442 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.620841+0000 mon.smithi097 (mon.0) 443 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.621437+0000 mon.smithi097 (mon.0) 444 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.622708+0000 mon.smithi097 (mon.0) 445 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.745267+0000 mon.smithi193 (mon.1) 15 : audit [INF] from='osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T06:51:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:12 smithi097 bash[24958]: audit 2023-11-17T06:51:11.746469+0000 mon.smithi097 (mon.0) 446 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:09.387210+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:09.387271+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:10.616457+0000 mon.smithi097 (mon.0) 429 : cluster [INF] osd.1 [v2:172.21.15.97:6802/3352242368,v1:172.21.15.97:6803/3352242368] boot 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:10.616487+0000 mon.smithi097 (mon.0) 430 : cluster [DBG] osdmap e22: 8 total, 3 up, 8 in 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.616979+0000 mon.smithi097 (mon.0) 431 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.617248+0000 mon.smithi097 (mon.0) 432 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.617447+0000 mon.smithi097 (mon.0) 433 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.617659+0000 mon.smithi097 (mon.0) 434 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.617826+0000 mon.smithi097 (mon.0) 435 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.618025+0000 mon.smithi097 (mon.0) 436 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:10.618270+0000 mon.smithi097 (mon.0) 437 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:11.319219+0000 mgr.smithi097.swvonl (mgr.14180) 117 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.618500+0000 mon.smithi097 (mon.0) 438 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: cluster 2023-11-17T06:51:11.618569+0000 mon.smithi097 (mon.0) 439 : cluster [DBG] osdmap e23: 8 total, 3 up, 8 in 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.618835+0000 mon.smithi097 (mon.0) 440 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.619684+0000 mon.smithi097 (mon.0) 441 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.620211+0000 mon.smithi097 (mon.0) 442 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.620841+0000 mon.smithi097 (mon.0) 443 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.621437+0000 mon.smithi097 (mon.0) 444 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.622708+0000 mon.smithi097 (mon.0) 445 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2023-11-17T06:51:12.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.745267+0000 mon.smithi193 (mon.1) 15 : audit [INF] from='osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T06:51:12.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:12 smithi193 bash[28559]: audit 2023-11-17T06:51:11.746469+0000 mon.smithi097 (mon.0) 446 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-17T06:51:13.635 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.625755+0000 mon.smithi097 (mon.0) 447 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.625880+0000 mon.smithi097 (mon.0) 448 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: cluster 2023-11-17T06:51:12.626011+0000 mon.smithi097 (mon.0) 449 : cluster [DBG] osdmap e24: 8 total, 3 up, 8 in 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.626322+0000 mon.smithi097 (mon.0) 450 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.626881+0000 mon.smithi193 (mon.1) 16 : audit [INF] from='osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.627048+0000 mon.smithi097 (mon.0) 451 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:13.636 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.627387+0000 mon.smithi097 (mon.0) 452 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:14.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.627850+0000 mon.smithi097 (mon.0) 453 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:14.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.628326+0000 mon.smithi097 (mon.0) 454 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:14.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:13 smithi097 bash[24958]: audit 2023-11-17T06:51:12.628778+0000 mon.smithi097 (mon.0) 455 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.625755+0000 mon.smithi097 (mon.0) 447 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.625880+0000 mon.smithi097 (mon.0) 448 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: cluster 2023-11-17T06:51:12.626011+0000 mon.smithi097 (mon.0) 449 : cluster [DBG] osdmap e24: 8 total, 3 up, 8 in 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.626322+0000 mon.smithi097 (mon.0) 450 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.626881+0000 mon.smithi193 (mon.1) 16 : audit [INF] from='osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.627048+0000 mon.smithi097 (mon.0) 451 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.627387+0000 mon.smithi097 (mon.0) 452 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.627850+0000 mon.smithi097 (mon.0) 453 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.628326+0000 mon.smithi097 (mon.0) 454 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:13 smithi193 bash[28559]: audit 2023-11-17T06:51:12.628778+0000 mon.smithi097 (mon.0) 455 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:14.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: cluster 2023-11-17T06:51:13.319721+0000 mgr.smithi097.swvonl (mgr.14180) 118 : cluster [DBG] pgmap v89: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.631203+0000 mon.smithi097 (mon.0) 456 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: cluster 2023-11-17T06:51:13.631268+0000 mon.smithi097 (mon.0) 457 : cluster [DBG] osdmap e25: 8 total, 3 up, 8 in 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.631626+0000 mon.smithi097 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.632082+0000 mon.smithi097 (mon.0) 459 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.632354+0000 mon.smithi097 (mon.0) 460 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.632555+0000 mon.smithi097 (mon.0) 461 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.632807+0000 mon.smithi097 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:14 smithi097 bash[24958]: audit 2023-11-17T06:51:13.637659+0000 mon.smithi097 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: cluster 2023-11-17T06:51:13.319721+0000 mgr.smithi097.swvonl (mgr.14180) 118 : cluster [DBG] pgmap v89: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.631203+0000 mon.smithi097 (mon.0) 456 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: cluster 2023-11-17T06:51:13.631268+0000 mon.smithi097 (mon.0) 457 : cluster [DBG] osdmap e25: 8 total, 3 up, 8 in 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.631626+0000 mon.smithi097 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.632082+0000 mon.smithi097 (mon.0) 459 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.632354+0000 mon.smithi097 (mon.0) 460 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.632555+0000 mon.smithi097 (mon.0) 461 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.632807+0000 mon.smithi097 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:14 smithi193 bash[28559]: audit 2023-11-17T06:51:13.637659+0000 mon.smithi097 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: cluster 2023-11-17T06:51:12.775548+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: cluster 2023-11-17T06:51:12.775623+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: cluster 2023-11-17T06:51:14.631581+0000 mon.smithi097 (mon.0) 464 : cluster [INF] osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517] boot 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: cluster 2023-11-17T06:51:14.631611+0000 mon.smithi097 (mon.0) 465 : cluster [DBG] osdmap e26: 8 total, 4 up, 8 in 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: audit 2023-11-17T06:51:14.632151+0000 mon.smithi097 (mon.0) 466 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: audit 2023-11-17T06:51:14.632634+0000 mon.smithi097 (mon.0) 467 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:15.765 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: audit 2023-11-17T06:51:14.633168+0000 mon.smithi097 (mon.0) 468 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:15.766 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: audit 2023-11-17T06:51:14.633578+0000 mon.smithi097 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:15.766 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:15 smithi097 bash[24958]: audit 2023-11-17T06:51:14.633970+0000 mon.smithi097 (mon.0) 470 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: cluster 2023-11-17T06:51:12.775548+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: cluster 2023-11-17T06:51:12.775623+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: cluster 2023-11-17T06:51:14.631581+0000 mon.smithi097 (mon.0) 464 : cluster [INF] osd.4 [v2:172.21.15.193:6816/2167424517,v1:172.21.15.193:6817/2167424517] boot 2023-11-17T06:51:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: cluster 2023-11-17T06:51:14.631611+0000 mon.smithi097 (mon.0) 465 : cluster [DBG] osdmap e26: 8 total, 4 up, 8 in 2023-11-17T06:51:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: audit 2023-11-17T06:51:14.632151+0000 mon.smithi097 (mon.0) 466 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:16.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: audit 2023-11-17T06:51:14.632634+0000 mon.smithi097 (mon.0) 467 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-17T06:51:16.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: audit 2023-11-17T06:51:14.633168+0000 mon.smithi097 (mon.0) 468 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:16.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: audit 2023-11-17T06:51:14.633578+0000 mon.smithi097 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:16.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:15 smithi193 bash[28559]: audit 2023-11-17T06:51:14.633970+0000 mon.smithi097 (mon.0) 470 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:16.248 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: cluster 2023-11-17T06:51:15.320333+0000 mgr.smithi097.swvonl (mgr.14180) 119 : cluster [DBG] pgmap v92: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: cluster 2023-11-17T06:51:15.641969+0000 mon.smithi097 (mon.0) 471 : cluster [DBG] osdmap e27: 8 total, 4 up, 8 in 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:15.642689+0000 mon.smithi097 (mon.0) 472 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:15.643238+0000 mon.smithi097 (mon.0) 473 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:15.643745+0000 mon.smithi097 (mon.0) 474 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:16.903 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:15.647732+0000 mon.smithi097 (mon.0) 475 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:16.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:16.045115+0000 mon.smithi097 (mon.0) 476 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:16.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:16.046274+0000 mon.smithi097 (mon.0) 477 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-17T06:51:16.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:16.047352+0000 mon.smithi097 (mon.0) 478 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:16.904 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:16 smithi193 bash[28559]: audit 2023-11-17T06:51:16.247223+0000 mon.smithi097 (mon.0) 479 : audit [DBG] from='client.? 172.21.15.97:0/1692252751' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:17.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: cluster 2023-11-17T06:51:15.320333+0000 mgr.smithi097.swvonl (mgr.14180) 119 : cluster [DBG] pgmap v92: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:17.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: cluster 2023-11-17T06:51:15.641969+0000 mon.smithi097 (mon.0) 471 : cluster [DBG] osdmap e27: 8 total, 4 up, 8 in 2023-11-17T06:51:17.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:15.642689+0000 mon.smithi097 (mon.0) 472 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:17.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:15.643238+0000 mon.smithi097 (mon.0) 473 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:17.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:15.643745+0000 mon.smithi097 (mon.0) 474 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:17.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:15.647732+0000 mon.smithi097 (mon.0) 475 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:17.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:16.045115+0000 mon.smithi097 (mon.0) 476 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:17.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:16.046274+0000 mon.smithi097 (mon.0) 477 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-17T06:51:17.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:16.047352+0000 mon.smithi097 (mon.0) 478 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:51:17.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:16 smithi097 bash[24958]: audit 2023-11-17T06:51:16.247223+0000 mon.smithi097 (mon.0) 479 : audit [DBG] from='client.? 172.21.15.97:0/1692252751' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:18.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:17 smithi097 bash[24958]: cephadm 2023-11-17T06:51:16.048224+0000 mgr.smithi097.swvonl (mgr.14180) 120 : cephadm [INF] Deploying daemon osd.7 on smithi097 2023-11-17T06:51:18.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:17 smithi097 bash[24958]: audit 2023-11-17T06:51:16.794524+0000 mon.smithi193 (mon.1) 17 : audit [INF] from='osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T06:51:18.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:17 smithi097 bash[24958]: audit 2023-11-17T06:51:16.795683+0000 mon.smithi097 (mon.0) 480 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T06:51:18.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:17 smithi097 bash[24958]: audit 2023-11-17T06:51:17.364081+0000 mon.smithi097 (mon.0) 481 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-17T06:51:18.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:17 smithi193 bash[28559]: cephadm 2023-11-17T06:51:16.048224+0000 mgr.smithi097.swvonl (mgr.14180) 120 : cephadm [INF] Deploying daemon osd.7 on smithi097 2023-11-17T06:51:18.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:17 smithi193 bash[28559]: audit 2023-11-17T06:51:16.794524+0000 mon.smithi193 (mon.1) 17 : audit [INF] from='osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T06:51:18.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:17 smithi193 bash[28559]: audit 2023-11-17T06:51:16.795683+0000 mon.smithi097 (mon.0) 480 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-17T06:51:18.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:17 smithi193 bash[28559]: audit 2023-11-17T06:51:17.364081+0000 mon.smithi097 (mon.0) 481 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-17T06:51:18.058 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":27,"num_osds":8,"num_up_osds":4,"osd_up_since":1700203874,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: cluster 2023-11-17T06:51:17.320878+0000 mgr.smithi097.swvonl (mgr.14180) 121 : cluster [DBG] pgmap v94: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.655291+0000 mon.smithi097 (mon.0) 482 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.655355+0000 mon.smithi097 (mon.0) 483 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: cluster 2023-11-17T06:51:17.655407+0000 mon.smithi097 (mon.0) 484 : cluster [DBG] osdmap e28: 8 total, 4 up, 8 in 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.655646+0000 mon.smithi097 (mon.0) 485 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.656233+0000 mon.smithi193 (mon.1) 18 : audit [INF] from='osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.656285+0000 mon.smithi097 (mon.0) 486 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.656663+0000 mon.smithi097 (mon.0) 487 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.657139+0000 mon.smithi097 (mon.0) 488 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.657556+0000 mon.smithi097 (mon.0) 489 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:18 smithi193 bash[28559]: audit 2023-11-17T06:51:17.657916+0000 mon.smithi097 (mon.0) 490 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:19.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: cluster 2023-11-17T06:51:17.320878+0000 mgr.smithi097.swvonl (mgr.14180) 121 : cluster [DBG] pgmap v94: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.655291+0000 mon.smithi097 (mon.0) 482 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.655355+0000 mon.smithi097 (mon.0) 483 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: cluster 2023-11-17T06:51:17.655407+0000 mon.smithi097 (mon.0) 484 : cluster [DBG] osdmap e28: 8 total, 4 up, 8 in 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.655646+0000 mon.smithi097 (mon.0) 485 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.656233+0000 mon.smithi193 (mon.1) 18 : audit [INF] from='osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.656285+0000 mon.smithi097 (mon.0) 486 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.656663+0000 mon.smithi097 (mon.0) 487 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.657139+0000 mon.smithi097 (mon.0) 488 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.657556+0000 mon.smithi097 (mon.0) 489 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2023-11-17T06:51:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:18 smithi097 bash[24958]: audit 2023-11-17T06:51:17.657916+0000 mon.smithi097 (mon.0) 490 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:19.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.657741+0000 mon.smithi097 (mon.0) 491 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.657832+0000 mon.smithi097 (mon.0) 492 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: cluster 2023-11-17T06:51:18.657909+0000 mon.smithi097 (mon.0) 493 : cluster [DBG] osdmap e29: 8 total, 4 up, 8 in 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.658638+0000 mon.smithi097 (mon.0) 494 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.659115+0000 mon.smithi097 (mon.0) 495 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.659777+0000 mon.smithi097 (mon.0) 496 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.660088+0000 mon.smithi097 (mon.0) 497 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.665882+0000 mon.smithi097 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:18.666222+0000 mon.smithi097 (mon.0) 499 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: cluster 2023-11-17T06:51:19.660909+0000 mon.smithi097 (mon.0) 500 : cluster [INF] osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819] boot 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: cluster 2023-11-17T06:51:19.660955+0000 mon.smithi097 (mon.0) 501 : cluster [INF] osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074] boot 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: cluster 2023-11-17T06:51:19.660989+0000 mon.smithi097 (mon.0) 502 : cluster [DBG] osdmap e30: 8 total, 6 up, 8 in 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:19.661116+0000 mon.smithi097 (mon.0) 503 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:19 smithi193 bash[28559]: audit 2023-11-17T06:51:19.661595+0000 mon.smithi097 (mon.0) 504 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.657741+0000 mon.smithi097 (mon.0) 491 : audit [INF] from='osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.657832+0000 mon.smithi097 (mon.0) 492 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: cluster 2023-11-17T06:51:18.657909+0000 mon.smithi097 (mon.0) 493 : cluster [DBG] osdmap e29: 8 total, 4 up, 8 in 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.658638+0000 mon.smithi097 (mon.0) 494 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.659115+0000 mon.smithi097 (mon.0) 495 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.659777+0000 mon.smithi097 (mon.0) 496 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.660088+0000 mon.smithi097 (mon.0) 497 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.665882+0000 mon.smithi097 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:18.666222+0000 mon.smithi097 (mon.0) 499 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: cluster 2023-11-17T06:51:19.660909+0000 mon.smithi097 (mon.0) 500 : cluster [INF] osd.3 [v2:172.21.15.97:6810/1152479819,v1:172.21.15.97:6811/1152479819] boot 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: cluster 2023-11-17T06:51:19.660955+0000 mon.smithi097 (mon.0) 501 : cluster [INF] osd.6 [v2:172.21.15.193:6824/2704435074,v1:172.21.15.193:6825/2704435074] boot 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: cluster 2023-11-17T06:51:19.660989+0000 mon.smithi097 (mon.0) 502 : cluster [DBG] osdmap e30: 8 total, 6 up, 8 in 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:19.661116+0000 mon.smithi097 (mon.0) 503 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-17T06:51:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:19 smithi097 bash[24958]: audit 2023-11-17T06:51:19.661595+0000 mon.smithi097 (mon.0) 504 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:17.833363+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:17.833439+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:18.379615+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:18.379770+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:19.321365+0000 mgr.smithi097.swvonl (mgr.14180) 122 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: audit 2023-11-17T06:51:19.662405+0000 mon.smithi097 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: audit 2023-11-17T06:51:19.662684+0000 mon.smithi097 (mon.0) 506 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: audit 2023-11-17T06:51:20.207214+0000 mon.smithi097 (mon.0) 507 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: cluster 2023-11-17T06:51:20.667145+0000 mon.smithi097 (mon.0) 508 : cluster [DBG] osdmap e31: 8 total, 6 up, 8 in 2023-11-17T06:51:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: audit 2023-11-17T06:51:20.667339+0000 mon.smithi097 (mon.0) 509 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:20 smithi097 bash[24958]: audit 2023-11-17T06:51:20.667848+0000 mon.smithi097 (mon.0) 510 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:17.833363+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:17.833439+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:18.379615+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:18.379770+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:19.321365+0000 mgr.smithi097.swvonl (mgr.14180) 122 : cluster [DBG] pgmap v97: 1 pgs: 1 peering; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: audit 2023-11-17T06:51:19.662405+0000 mon.smithi097 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: audit 2023-11-17T06:51:19.662684+0000 mon.smithi097 (mon.0) 506 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: audit 2023-11-17T06:51:20.207214+0000 mon.smithi097 (mon.0) 507 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: cluster 2023-11-17T06:51:20.667145+0000 mon.smithi097 (mon.0) 508 : cluster [DBG] osdmap e31: 8 total, 6 up, 8 in 2023-11-17T06:51:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: audit 2023-11-17T06:51:20.667339+0000 mon.smithi097 (mon.0) 509 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:21.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:20 smithi193 bash[28559]: audit 2023-11-17T06:51:20.667848+0000 mon.smithi097 (mon.0) 510 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:22.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:22 smithi097 bash[24958]: cluster 2023-11-17T06:51:21.321858+0000 mgr.smithi097.swvonl (mgr.14180) 123 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:22.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:22 smithi097 bash[24958]: cluster 2023-11-17T06:51:21.666133+0000 mon.smithi097 (mon.0) 511 : cluster [DBG] osdmap e32: 8 total, 6 up, 8 in 2023-11-17T06:51:22.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:22 smithi097 bash[24958]: audit 2023-11-17T06:51:21.666338+0000 mon.smithi097 (mon.0) 512 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:22.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:22 smithi097 bash[24958]: audit 2023-11-17T06:51:21.666764+0000 mon.smithi097 (mon.0) 513 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:22.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:22 smithi193 bash[28559]: cluster 2023-11-17T06:51:21.321858+0000 mgr.smithi097.swvonl (mgr.14180) 123 : cluster [DBG] pgmap v100: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:22 smithi193 bash[28559]: cluster 2023-11-17T06:51:21.666133+0000 mon.smithi097 (mon.0) 511 : cluster [DBG] osdmap e32: 8 total, 6 up, 8 in 2023-11-17T06:51:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:22 smithi193 bash[28559]: audit 2023-11-17T06:51:21.666338+0000 mon.smithi097 (mon.0) 512 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:22 smithi193 bash[28559]: audit 2023-11-17T06:51:21.666764+0000 mon.smithi097 (mon.0) 513 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:24.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:24 smithi097 bash[24958]: cluster 2023-11-17T06:51:23.322209+0000 mgr.smithi097.swvonl (mgr.14180) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:24.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:24 smithi193 bash[28559]: cluster 2023-11-17T06:51:23.322209+0000 mgr.smithi097.swvonl (mgr.14180) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:25.602 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:26.480 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:26 smithi097 bash[24958]: audit 2023-11-17T06:51:25.068042+0000 mon.smithi097 (mon.0) 514 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:26.480 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:26 smithi097 bash[24958]: cluster 2023-11-17T06:51:25.322629+0000 mgr.smithi097.swvonl (mgr.14180) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:26.480 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:26 smithi097 bash[24958]: audit 2023-11-17T06:51:25.601407+0000 mon.smithi097 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.97:0/3467943906' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:26.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:26 smithi193 bash[28559]: audit 2023-11-17T06:51:25.068042+0000 mon.smithi097 (mon.0) 514 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:26.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:26 smithi193 bash[28559]: cluster 2023-11-17T06:51:25.322629+0000 mgr.smithi097.swvonl (mgr.14180) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:26.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:26 smithi193 bash[28559]: audit 2023-11-17T06:51:25.601407+0000 mon.smithi097 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.97:0/3467943906' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:26.995 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":32,"num_osds":8,"num_up_osds":6,"osd_up_since":1700203879,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:51:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:27 smithi097 bash[24958]: audit 2023-11-17T06:51:26.482542+0000 mon.smithi097 (mon.0) 516 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-17T06:51:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:27 smithi193 bash[28559]: audit 2023-11-17T06:51:26.482542+0000 mon.smithi097 (mon.0) 516 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-17T06:51:27.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: audit 2023-11-17T06:51:27.094212+0000 mon.smithi097 (mon.0) 517 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: cluster 2023-11-17T06:51:27.094380+0000 mon.smithi097 (mon.0) 518 : cluster [DBG] osdmap e33: 8 total, 6 up, 8 in 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: audit 2023-11-17T06:51:27.094590+0000 mon.smithi097 (mon.0) 519 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: audit 2023-11-17T06:51:27.095053+0000 mon.smithi097 (mon.0) 520 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: audit 2023-11-17T06:51:27.095319+0000 mon.smithi097 (mon.0) 521 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:28.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:28 smithi097 bash[24958]: cluster 2023-11-17T06:51:27.323080+0000 mgr.smithi097.swvonl (mgr.14180) 126 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: audit 2023-11-17T06:51:27.094212+0000 mon.smithi097 (mon.0) 517 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: cluster 2023-11-17T06:51:27.094380+0000 mon.smithi097 (mon.0) 518 : cluster [DBG] osdmap e33: 8 total, 6 up, 8 in 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: audit 2023-11-17T06:51:27.094590+0000 mon.smithi097 (mon.0) 519 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: audit 2023-11-17T06:51:27.095053+0000 mon.smithi097 (mon.0) 520 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: audit 2023-11-17T06:51:27.095319+0000 mon.smithi097 (mon.0) 521 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:28.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:28 smithi193 bash[28559]: cluster 2023-11-17T06:51:27.323080+0000 mgr.smithi097.swvonl (mgr.14180) 126 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: audit 2023-11-17T06:51:28.094665+0000 mon.smithi097 (mon.0) 522 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: cluster 2023-11-17T06:51:28.094729+0000 mon.smithi097 (mon.0) 523 : cluster [DBG] osdmap e34: 8 total, 6 up, 8 in 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: audit 2023-11-17T06:51:28.095156+0000 mon.smithi097 (mon.0) 524 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: audit 2023-11-17T06:51:28.095700+0000 mon.smithi097 (mon.0) 525 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: audit 2023-11-17T06:51:28.099986+0000 mon.smithi097 (mon.0) 526 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:29 smithi097 bash[24958]: audit 2023-11-17T06:51:29.099394+0000 mon.smithi097 (mon.0) 527 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: audit 2023-11-17T06:51:28.094665+0000 mon.smithi097 (mon.0) 522 : audit [INF] from='osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: cluster 2023-11-17T06:51:28.094729+0000 mon.smithi097 (mon.0) 523 : cluster [DBG] osdmap e34: 8 total, 6 up, 8 in 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: audit 2023-11-17T06:51:28.095156+0000 mon.smithi097 (mon.0) 524 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: audit 2023-11-17T06:51:28.095700+0000 mon.smithi097 (mon.0) 525 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: audit 2023-11-17T06:51:28.099986+0000 mon.smithi097 (mon.0) 526 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:29 smithi193 bash[28559]: audit 2023-11-17T06:51:29.099394+0000 mon.smithi097 (mon.0) 527 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: cluster 2023-11-17T06:51:27.442798+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: cluster 2023-11-17T06:51:27.442884+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: cluster 2023-11-17T06:51:29.113090+0000 mon.smithi097 (mon.0) 528 : cluster [INF] osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634] boot 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: cluster 2023-11-17T06:51:29.113129+0000 mon.smithi097 (mon.0) 529 : cluster [DBG] osdmap e35: 8 total, 7 up, 8 in 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: audit 2023-11-17T06:51:29.113286+0000 mon.smithi097 (mon.0) 530 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: audit 2023-11-17T06:51:29.113694+0000 mon.smithi097 (mon.0) 531 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:30.390 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:30 smithi097 bash[24958]: cluster 2023-11-17T06:51:29.323647+0000 mgr.smithi097.swvonl (mgr.14180) 127 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: cluster 2023-11-17T06:51:27.442798+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: cluster 2023-11-17T06:51:27.442884+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: cluster 2023-11-17T06:51:29.113090+0000 mon.smithi097 (mon.0) 528 : cluster [INF] osd.5 [v2:172.21.15.97:6818/3902830634,v1:172.21.15.97:6819/3902830634] boot 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: cluster 2023-11-17T06:51:29.113129+0000 mon.smithi097 (mon.0) 529 : cluster [DBG] osdmap e35: 8 total, 7 up, 8 in 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: audit 2023-11-17T06:51:29.113286+0000 mon.smithi097 (mon.0) 530 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: audit 2023-11-17T06:51:29.113694+0000 mon.smithi097 (mon.0) 531 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:30 smithi193 bash[28559]: cluster 2023-11-17T06:51:29.323647+0000 mgr.smithi097.swvonl (mgr.14180) 127 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-11-17T06:51:31.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:31 smithi097 bash[24958]: cluster 2023-11-17T06:51:30.121206+0000 mon.smithi097 (mon.0) 532 : cluster [DBG] osdmap e36: 8 total, 7 up, 8 in 2023-11-17T06:51:31.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:31 smithi097 bash[24958]: audit 2023-11-17T06:51:30.121402+0000 mon.smithi097 (mon.0) 533 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:31.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:31 smithi193 bash[28559]: cluster 2023-11-17T06:51:30.121206+0000 mon.smithi097 (mon.0) 532 : cluster [DBG] osdmap e36: 8 total, 7 up, 8 in 2023-11-17T06:51:31.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:31 smithi193 bash[28559]: audit 2023-11-17T06:51:30.121402+0000 mon.smithi097 (mon.0) 533 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:32.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:32 smithi097 bash[24958]: cluster 2023-11-17T06:51:31.324014+0000 mgr.smithi097.swvonl (mgr.14180) 128 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:32.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:32 smithi193 bash[28559]: cluster 2023-11-17T06:51:31.324014+0000 mgr.smithi097.swvonl (mgr.14180) 128 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:33.007 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:33.309 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:33 smithi097 bash[24958]: audit 2023-11-17T06:51:33.005373+0000 mon.smithi097 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.97:0/1244816205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:33.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:33 smithi193 bash[28559]: audit 2023-11-17T06:51:33.005373+0000 mon.smithi097 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.97:0/1244816205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:33.957 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":36,"num_osds":8,"num_up_osds":7,"osd_up_since":1700203889,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:51:34.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:34 smithi097 bash[24958]: cluster 2023-11-17T06:51:33.324548+0000 mgr.smithi097.swvonl (mgr.14180) 129 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:34 smithi193 bash[28559]: cluster 2023-11-17T06:51:33.324548+0000 mgr.smithi097.swvonl (mgr.14180) 129 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:34.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd stat -f json 2023-11-17T06:51:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:35 smithi097 bash[24958]: audit 2023-11-17T06:51:34.572289+0000 mon.smithi097 (mon.0) 535 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-17T06:51:35.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:35 smithi193 bash[28559]: audit 2023-11-17T06:51:34.572289+0000 mon.smithi097 (mon.0) 535 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-17T06:51:36.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:36 smithi097 bash[24958]: audit 2023-11-17T06:51:35.149756+0000 mon.smithi097 (mon.0) 536 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-17T06:51:36.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:36 smithi097 bash[24958]: cluster 2023-11-17T06:51:35.149826+0000 mon.smithi097 (mon.0) 537 : cluster [DBG] osdmap e37: 8 total, 7 up, 8 in 2023-11-17T06:51:36.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:36 smithi097 bash[24958]: audit 2023-11-17T06:51:35.149987+0000 mon.smithi097 (mon.0) 538 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:36.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:36 smithi097 bash[24958]: audit 2023-11-17T06:51:35.150712+0000 mon.smithi097 (mon.0) 539 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:36.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:36 smithi097 bash[24958]: cluster 2023-11-17T06:51:35.325030+0000 mgr.smithi097.swvonl (mgr.14180) 130 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:36.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:36 smithi193 bash[28559]: audit 2023-11-17T06:51:35.149756+0000 mon.smithi097 (mon.0) 536 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-17T06:51:36.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:36 smithi193 bash[28559]: cluster 2023-11-17T06:51:35.149826+0000 mon.smithi097 (mon.0) 537 : cluster [DBG] osdmap e37: 8 total, 7 up, 8 in 2023-11-17T06:51:36.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:36 smithi193 bash[28559]: audit 2023-11-17T06:51:35.149987+0000 mon.smithi097 (mon.0) 538 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:36.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:36 smithi193 bash[28559]: audit 2023-11-17T06:51:35.150712+0000 mon.smithi097 (mon.0) 539 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2023-11-17T06:51:36.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:36 smithi193 bash[28559]: cluster 2023-11-17T06:51:35.325030+0000 mgr.smithi097.swvonl (mgr.14180) 130 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:37.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:37 smithi097 bash[24958]: audit 2023-11-17T06:51:36.151580+0000 mon.smithi097 (mon.0) 540 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:37.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:37 smithi097 bash[24958]: cluster 2023-11-17T06:51:36.151641+0000 mon.smithi097 (mon.0) 541 : cluster [DBG] osdmap e38: 8 total, 7 up, 8 in 2023-11-17T06:51:37.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:37 smithi097 bash[24958]: audit 2023-11-17T06:51:36.152045+0000 mon.smithi097 (mon.0) 542 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:37.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:37 smithi097 bash[24958]: audit 2023-11-17T06:51:36.158433+0000 mon.smithi097 (mon.0) 543 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:37.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:37 smithi193 bash[28559]: audit 2023-11-17T06:51:36.151580+0000 mon.smithi097 (mon.0) 540 : audit [INF] from='osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2023-11-17T06:51:37.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:37 smithi193 bash[28559]: cluster 2023-11-17T06:51:36.151641+0000 mon.smithi097 (mon.0) 541 : cluster [DBG] osdmap e38: 8 total, 7 up, 8 in 2023-11-17T06:51:37.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:37 smithi193 bash[28559]: audit 2023-11-17T06:51:36.152045+0000 mon.smithi097 (mon.0) 542 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:37.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:37 smithi193 bash[28559]: audit 2023-11-17T06:51:36.158433+0000 mon.smithi097 (mon.0) 543 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: cluster 2023-11-17T06:51:35.545722+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: cluster 2023-11-17T06:51:35.545849+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: cluster 2023-11-17T06:51:37.161548+0000 mon.smithi097 (mon.0) 544 : cluster [INF] osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798] boot 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: cluster 2023-11-17T06:51:37.161649+0000 mon.smithi097 (mon.0) 545 : cluster [DBG] osdmap e39: 8 total, 8 up, 8 in 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: audit 2023-11-17T06:51:37.162223+0000 mon.smithi097 (mon.0) 546 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:38 smithi097 bash[24958]: cluster 2023-11-17T06:51:37.325614+0000 mgr.smithi097.swvonl (mgr.14180) 131 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: cluster 2023-11-17T06:51:35.545722+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: cluster 2023-11-17T06:51:35.545849+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: cluster 2023-11-17T06:51:37.161548+0000 mon.smithi097 (mon.0) 544 : cluster [INF] osd.7 [v2:172.21.15.97:6826/2156009798,v1:172.21.15.97:6827/2156009798] boot 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: cluster 2023-11-17T06:51:37.161649+0000 mon.smithi097 (mon.0) 545 : cluster [DBG] osdmap e39: 8 total, 8 up, 8 in 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: audit 2023-11-17T06:51:37.162223+0000 mon.smithi097 (mon.0) 546 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-17T06:51:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:38 smithi193 bash[28559]: cluster 2023-11-17T06:51:37.325614+0000 mgr.smithi097.swvonl (mgr.14180) 131 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2023-11-17T06:51:39.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:39 smithi097 bash[24958]: cluster 2023-11-17T06:51:38.170006+0000 mon.smithi097 (mon.0) 547 : cluster [DBG] osdmap e40: 8 total, 8 up, 8 in 2023-11-17T06:51:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:39 smithi193 bash[28559]: cluster 2023-11-17T06:51:38.170006+0000 mon.smithi097 (mon.0) 547 : cluster [DBG] osdmap e40: 8 total, 8 up, 8 in 2023-11-17T06:51:40.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:40 smithi097 bash[24958]: cluster 2023-11-17T06:51:39.182005+0000 mon.smithi097 (mon.0) 548 : cluster [DBG] osdmap e41: 8 total, 8 up, 8 in 2023-11-17T06:51:40.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:40 smithi097 bash[24958]: cluster 2023-11-17T06:51:39.326349+0000 mgr.smithi097.swvonl (mgr.14180) 132 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:40.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:40 smithi193 bash[28559]: cluster 2023-11-17T06:51:39.182005+0000 mon.smithi097 (mon.0) 548 : cluster [DBG] osdmap e41: 8 total, 8 up, 8 in 2023-11-17T06:51:40.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:40 smithi193 bash[28559]: cluster 2023-11-17T06:51:39.326349+0000 mgr.smithi097.swvonl (mgr.14180) 132 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:40.593 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:41.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:41 smithi097 bash[24958]: audit 2023-11-17T06:51:40.592614+0000 mon.smithi097 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.97:0/2222324859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:41 smithi193 bash[28559]: audit 2023-11-17T06:51:40.592614+0000 mon.smithi097 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.97:0/2222324859' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-17T06:51:41.763 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":41,"num_osds":8,"num_up_osds":8,"osd_up_since":1700203897,"num_in_osds":8,"osd_in_since":1700203834,"num_remapped_pgs":0} 2023-11-17T06:51:41.763 INFO:tasks.cephadm:Setting up client nodes... 2023-11-17T06:51:41.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-17T06:51:42.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:42 smithi097 bash[24958]: cluster 2023-11-17T06:51:41.326733+0000 mgr.smithi097.swvonl (mgr.14180) 133 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:42.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:42 smithi193 bash[28559]: cluster 2023-11-17T06:51:41.326733+0000 mgr.smithi097.swvonl (mgr.14180) 133 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:44.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:43 smithi097 bash[24958]: audit 2023-11-17T06:51:42.723259+0000 mon.smithi097 (mon.0) 550 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:44.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:43 smithi097 bash[24958]: audit 2023-11-17T06:51:42.728669+0000 mon.smithi097 (mon.0) 551 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:51:44.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:43 smithi097 bash[24958]: cluster 2023-11-17T06:51:43.327262+0000 mgr.smithi097.swvonl (mgr.14180) 134 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:44.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:43 smithi193 bash[28559]: audit 2023-11-17T06:51:42.723259+0000 mon.smithi097 (mon.0) 550 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:44.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:43 smithi193 bash[28559]: audit 2023-11-17T06:51:42.728669+0000 mon.smithi097 (mon.0) 551 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:51:44.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:43 smithi193 bash[28559]: cluster 2023-11-17T06:51:43.327262+0000 mgr.smithi097.swvonl (mgr.14180) 134 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:45.714 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2023-11-17T06:51:45.714 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCBDVdly6UbKhAAiTSchxBLxNKAqsJrR3rrzA== 2023-11-17T06:51:46.510 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T06:51:46.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-17T06:51:46.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-17T06:51:46.533 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-17T06:51:46.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:46 smithi097 bash[24958]: cluster 2023-11-17T06:51:45.327868+0000 mgr.smithi097.swvonl (mgr.14180) 135 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:46.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:46 smithi097 bash[24958]: audit 2023-11-17T06:51:45.706077+0000 mon.smithi097 (mon.0) 552 : audit [INF] from='client.? 172.21.15.97:0/4071827065' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:46.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:46 smithi097 bash[24958]: audit 2023-11-17T06:51:45.712818+0000 mon.smithi097 (mon.0) 553 : audit [INF] from='client.? 172.21.15.97:0/4071827065' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T06:51:46.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:46 smithi193 bash[28559]: cluster 2023-11-17T06:51:45.327868+0000 mgr.smithi097.swvonl (mgr.14180) 135 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:46.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:46 smithi193 bash[28559]: audit 2023-11-17T06:51:45.706077+0000 mon.smithi097 (mon.0) 552 : audit [INF] from='client.? 172.21.15.97:0/4071827065' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:46.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:46 smithi193 bash[28559]: audit 2023-11-17T06:51:45.712818+0000 mon.smithi097 (mon.0) 553 : audit [INF] from='client.? 172.21.15.97:0/4071827065' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T06:51:48.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:48 smithi097 bash[24958]: cluster 2023-11-17T06:51:47.328450+0000 mgr.smithi097.swvonl (mgr.14180) 136 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:48 smithi097 bash[24958]: audit 2023-11-17T06:51:48.207421+0000 mon.smithi097 (mon.0) 554 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:48.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:48 smithi193 bash[28559]: cluster 2023-11-17T06:51:47.328450+0000 mgr.smithi097.swvonl (mgr.14180) 136 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:48.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:48 smithi193 bash[28559]: audit 2023-11-17T06:51:48.207421+0000 mon.smithi097 (mon.0) 554 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:49.665 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2023-11-17T06:51:49.666 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQCFDVdleTQJJxAA1ya0WpXb668HfVJWTACjFQ== 2023-11-17T06:51:50.471 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T06:51:50.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-11-17T06:51:50.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-17T06:51:50.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T06:51:50.506 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:50 smithi097 bash[24958]: audit 2023-11-17T06:51:49.239857+0000 mon.smithi097 (mon.0) 555 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:50.506 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:50 smithi097 bash[24958]: cluster 2023-11-17T06:51:49.329031+0000 mgr.smithi097.swvonl (mgr.14180) 137 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:50.506 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:50 smithi097 bash[24958]: audit 2023-11-17T06:51:49.653335+0000 mon.smithi193 (mon.1) 19 : audit [INF] from='client.? 172.21.15.193:0/1151340153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:50.506 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:50 smithi097 bash[24958]: audit 2023-11-17T06:51:49.654580+0000 mon.smithi097 (mon.0) 556 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:50.506 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:50 smithi097 bash[24958]: audit 2023-11-17T06:51:49.663054+0000 mon.smithi097 (mon.0) 557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T06:51:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:50 smithi193 bash[28559]: audit 2023-11-17T06:51:49.239857+0000 mon.smithi097 (mon.0) 555 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:50 smithi193 bash[28559]: cluster 2023-11-17T06:51:49.329031+0000 mgr.smithi097.swvonl (mgr.14180) 137 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:50 smithi193 bash[28559]: audit 2023-11-17T06:51:49.653335+0000 mon.smithi193 (mon.1) 19 : audit [INF] from='client.? 172.21.15.193:0/1151340153' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:50 smithi193 bash[28559]: audit 2023-11-17T06:51:49.654580+0000 mon.smithi097 (mon.0) 556 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-17T06:51:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:50 smithi193 bash[28559]: audit 2023-11-17T06:51:49.663054+0000 mon.smithi097 (mon.0) 557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-17T06:51:52.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:52 smithi097 bash[24958]: cluster 2023-11-17T06:51:51.329628+0000 mgr.smithi097.swvonl (mgr.14180) 138 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:52 smithi193 bash[28559]: cluster 2023-11-17T06:51:51.329628+0000 mgr.smithi097.swvonl (mgr.14180) 138 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:54.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:54 smithi097 bash[24958]: cluster 2023-11-17T06:51:53.330232+0000 mgr.smithi097.swvonl (mgr.14180) 139 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:54.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:54 smithi193 bash[28559]: cluster 2023-11-17T06:51:53.330232+0000 mgr.smithi097.swvonl (mgr.14180) 139 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:55.044 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:51:56.512 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:56 smithi097 bash[24958]: cluster 2023-11-17T06:51:55.330819+0000 mgr.smithi097.swvonl (mgr.14180) 140 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:56.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:56 smithi193 bash[28559]: cluster 2023-11-17T06:51:55.330819+0000 mgr.smithi097.swvonl (mgr.14180) 140 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:58.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:58 smithi097 bash[24958]: cluster 2023-11-17T06:51:57.331304+0000 mgr.smithi097.swvonl (mgr.14180) 141 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:58.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:58 smithi097 bash[24958]: audit 2023-11-17T06:51:58.221184+0000 mon.smithi097 (mon.0) 558 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:58 smithi193 bash[28559]: cluster 2023-11-17T06:51:57.331304+0000 mgr.smithi097.swvonl (mgr.14180) 141 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:51:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:58 smithi193 bash[28559]: audit 2023-11-17T06:51:58.221184+0000 mon.smithi097 (mon.0) 558 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:59.486 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: cephadm 2023-11-17T06:51:58.210839+0000 mgr.smithi097.swvonl (mgr.14180) 142 : cephadm [INF] Detected new or changed devices on smithi193 2023-11-17T06:51:59.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: audit 2023-11-17T06:51:58.515425+0000 mon.smithi097 (mon.0) 559 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.559 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: audit 2023-11-17T06:51:58.516945+0000 mon.smithi097 (mon.0) 560 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.559 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: audit 2023-11-17T06:51:58.517970+0000 mon.smithi097 (mon.0) 561 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.559 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: audit 2023-11-17T06:51:58.519010+0000 mon.smithi097 (mon.0) 562 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.559 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: cephadm 2023-11-17T06:51:58.519777+0000 mgr.smithi097.swvonl (mgr.14180) 143 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4028M 2023-11-17T06:51:59.559 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:51:59 smithi097 bash[24958]: audit 2023-11-17T06:51:58.526451+0000 mon.smithi097 (mon.0) 563 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: cephadm 2023-11-17T06:51:58.210839+0000 mgr.smithi097.swvonl (mgr.14180) 142 : cephadm [INF] Detected new or changed devices on smithi193 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: audit 2023-11-17T06:51:58.515425+0000 mon.smithi097 (mon.0) 559 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: audit 2023-11-17T06:51:58.516945+0000 mon.smithi097 (mon.0) 560 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: audit 2023-11-17T06:51:58.517970+0000 mon.smithi097 (mon.0) 561 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: audit 2023-11-17T06:51:58.519010+0000 mon.smithi097 (mon.0) 562 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: cephadm 2023-11-17T06:51:58.519777+0000 mgr.smithi097.swvonl (mgr.14180) 143 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4028M 2023-11-17T06:51:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:51:59 smithi193 bash[28559]: audit 2023-11-17T06:51:58.526451+0000 mon.smithi097 (mon.0) 563 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:51:59.904 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:51:59.904 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":41,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","created":"2023-11-17T06:46:29.288434+0000","modified":"2023-11-17T06:51:39.172152+0000","last_up_change":"2023-11-17T06:51:37.154487+0000","last_in_change":"2023-11-17T06:50:34.925718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T06:51:11.076187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8e0cefb6-3c13-4136-bee5-9a4a3589ccd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6801","nonce":2497341844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6803","nonce":2497341844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6807","nonce":2497341844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6805","nonce":2497341844}]},"public_addr":"172.21.15.193:6801/2497341844","cluster_addr":"172.21.15.193:6803/2497341844","heartbeat_back_addr":"172.21.15.193:6807/2497341844","heartbeat_front_addr":"172.21.15.193:6805/2497341844","state":["exists","up"]},{"osd":1,"uuid":"aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6803","nonce":3352242368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6805","nonce":3352242368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6809","nonce":3352242368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6807","nonce":3352242368}]},"public_addr":"172.21.15.97:6803/3352242368","cluster_addr":"172.21.15.97:6805/3352242368","heartbeat_back_addr":"172.21.15.97:6809/3352242368","heartbeat_front_addr":"172.21.15.97:6807/3352242368","state":["exists","up"]},{"osd":2,"uuid":"76c1caa8-14d7-43e5-bedf-44135cb7b231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6809","nonce":3825756302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6811","nonce":3825756302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6815","nonce":3825756302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6813","nonce":3825756302}]},"public_addr":"172.21.15.193:6809/3825756302","cluster_addr":"172.21.15.193:6811/3825756302","heartbeat_back_addr":"172.21.15.193:6815/3825756302","heartbeat_front_addr":"172.21.15.193:6813/3825756302","state":["exists","up"]},{"osd":3,"uuid":"9bb2d027-be64-4f25-92eb-0d72ee3f968c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6811","nonce":1152479819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6813","nonce":1152479819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6817","nonce":1152479819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6815","nonce":1152479819}]},"public_addr":"172.21.15.97:6811/1152479819","cluster_addr":"172.21.15.97:6813/1152479819","heartbeat_back_addr":"172.21.15.97:6817/1152479819","heartbeat_front_addr":"172.21.15.97:6815/1152479819","state":["exists","up"]},{"osd":4,"uuid":"8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6817","nonce":2167424517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6819","nonce":2167424517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6823","nonce":2167424517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6821","nonce":2167424517}]},"public_addr":"172.21.15.193:6817/2167424517","cluster_addr":"172.21.15.193:6819/2167424517","heartbeat_back_addr":"172.21.15.193:6823/2167424517","heartbeat_front_addr":"172.21.15.193:6821/2167424517","state":["exists","up"]},{"osd":5,"uuid":"7463f212-d4f5-427f-a4db-3835fce07e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6819","nonce":3902830634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6821","nonce":3902830634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6825","nonce":3902830634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6823","nonce":3902830634}]},"public_addr":"172.21.15.97:6819/3902830634","cluster_addr":"172.21.15.97:6821/3902830634","heartbeat_back_addr":"172.21.15.97:6825/3902830634","heartbeat_front_addr":"172.21.15.97:6823/3902830634","state":["exists","up"]},{"osd":6,"uuid":"2e7230c8-6a76-4ced-a43f-63071d3dde82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6825","nonce":2704435074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6827","nonce":2704435074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6831","nonce":2704435074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6829","nonce":2704435074}]},"public_addr":"172.21.15.193:6825/2704435074","cluster_addr":"172.21.15.193:6827/2704435074","heartbeat_back_addr":"172.21.15.193:6831/2704435074","heartbeat_front_addr":"172.21.15.193:6829/2704435074","state":["exists","up"]},{"osd":7,"uuid":"c4c1b9cb-3792-496d-b445-63a1e16c3823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6827","nonce":2156009798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6829","nonce":2156009798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6833","nonce":2156009798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6831","nonce":2156009798}]},"public_addr":"172.21.15.97:6827/2156009798","cluster_addr":"172.21.15.97:6829/2156009798","heartbeat_back_addr":"172.21.15.97:6833/2156009798","heartbeat_front_addr":"172.21.15.97:6831/2156009798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:01.239713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:09.387274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:06.883757+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:18.379782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:12.775627+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:27.442889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:17.833444+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:35.545858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/3905637356":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6800/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:0/1794480283":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/2127305343":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/404773595":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:6800/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/974873253":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6800/3584977603":"2023-11-18T06:46:57.468702+0000","172.21.15.97:0/396146213":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6801/3584977603":"2023-11-18T06:46:57.468702+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T06:52:00.732 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-11-17T06:51:11.076187+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2023-11-17T06:52:00.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd pool get device_health_metrics pg_num 2023-11-17T06:52:00.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:00 smithi097 bash[24958]: cluster 2023-11-17T06:51:59.331769+0000 mgr.smithi097.swvonl (mgr.14180) 144 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:00.789 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:00 smithi097 bash[24958]: audit 2023-11-17T06:51:59.901536+0000 mon.smithi097 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.97:0/1062160890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:00.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:00 smithi193 bash[28559]: cluster 2023-11-17T06:51:59.331769+0000 mgr.smithi097.swvonl (mgr.14180) 144 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:00.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:00 smithi193 bash[28559]: audit 2023-11-17T06:51:59.901536+0000 mon.smithi097 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.97:0/1062160890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:02 smithi097 bash[24958]: cluster 2023-11-17T06:52:01.332350+0000 mgr.smithi097.swvonl (mgr.14180) 145 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:02 smithi097 bash[24958]: audit 2023-11-17T06:52:01.973551+0000 mon.smithi097 (mon.0) 565 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:02.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:02 smithi193 bash[28559]: cluster 2023-11-17T06:52:01.332350+0000 mgr.smithi097.swvonl (mgr.14180) 145 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:02.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:02 smithi193 bash[28559]: audit 2023-11-17T06:52:01.973551+0000 mon.smithi097 (mon.0) 565 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: cephadm 2023-11-17T06:52:01.961913+0000 mgr.smithi097.swvonl (mgr.14180) 146 : cephadm [INF] Detected new or changed devices on smithi097 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.266549+0000 mon.smithi097 (mon.0) 566 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.267975+0000 mon.smithi097 (mon.0) 567 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.269156+0000 mon.smithi097 (mon.0) 568 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.270347+0000 mon.smithi097 (mon.0) 569 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: cephadm 2023-11-17T06:52:02.271251+0000 mgr.smithi097.swvonl (mgr.14180) 147 : cephadm [INF] Adjusting osd_memory_target on smithi097 to 3260M 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.278590+0000 mon.smithi097 (mon.0) 570 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.280310+0000 mon.smithi097 (mon.0) 571 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.281325+0000 mon.smithi097 (mon.0) 572 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.287332+0000 mon.smithi097 (mon.0) 573 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.294907+0000 mon.smithi097 (mon.0) 574 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:52:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.302420+0000 mon.smithi097 (mon.0) 575 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:52:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.305565+0000 mon.smithi097 (mon.0) 576 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:52:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.306021+0000 mon.smithi097 (mon.0) 577 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:03 smithi097 bash[24958]: audit 2023-11-17T06:52:02.307005+0000 mon.smithi097 (mon.0) 578 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: cephadm 2023-11-17T06:52:01.961913+0000 mgr.smithi097.swvonl (mgr.14180) 146 : cephadm [INF] Detected new or changed devices on smithi097 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.266549+0000 mon.smithi097 (mon.0) 566 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.267975+0000 mon.smithi097 (mon.0) 567 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.269156+0000 mon.smithi097 (mon.0) 568 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.270347+0000 mon.smithi097 (mon.0) 569 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T06:52:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: cephadm 2023-11-17T06:52:02.271251+0000 mgr.smithi097.swvonl (mgr.14180) 147 : cephadm [INF] Adjusting osd_memory_target on smithi097 to 3260M 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.278590+0000 mon.smithi097 (mon.0) 570 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.280310+0000 mon.smithi097 (mon.0) 571 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.281325+0000 mon.smithi097 (mon.0) 572 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.287332+0000 mon.smithi097 (mon.0) 573 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.294907+0000 mon.smithi097 (mon.0) 574 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.302420+0000 mon.smithi097 (mon.0) 575 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.305565+0000 mon.smithi097 (mon.0) 576 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.306021+0000 mon.smithi097 (mon.0) 577 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:03 smithi193 bash[28559]: audit 2023-11-17T06:52:02.307005+0000 mon.smithi097 (mon.0) 578 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:52:04.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:04 smithi097 bash[24958]: cluster 2023-11-17T06:52:03.332933+0000 mgr.smithi097.swvonl (mgr.14180) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:04.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:04 smithi193 bash[28559]: cluster 2023-11-17T06:52:03.332933+0000 mgr.smithi097.swvonl (mgr.14180) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:05.558 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:52:06.685 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:06 smithi097 bash[24958]: cluster 2023-11-17T06:52:05.333564+0000 mgr.smithi097.swvonl (mgr.14180) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:06.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:06 smithi193 bash[28559]: cluster 2023-11-17T06:52:05.333564+0000 mgr.smithi097.swvonl (mgr.14180) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:08.697 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:08 smithi097 bash[24958]: cluster 2023-11-17T06:52:07.334164+0000 mgr.smithi097.swvonl (mgr.14180) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:08.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:08 smithi193 bash[28559]: cluster 2023-11-17T06:52:07.334164+0000 mgr.smithi097.swvonl (mgr.14180) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:10 smithi097 bash[24958]: cluster 2023-11-17T06:52:09.334767+0000 mgr.smithi097.swvonl (mgr.14180) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:10.677 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2023-11-17T06:52:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:10 smithi193 bash[28559]: cluster 2023-11-17T06:52:09.334767+0000 mgr.smithi097.swvonl (mgr.14180) 151 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:11.689 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:11 smithi193 bash[28559]: audit 2023-11-17T06:52:10.675731+0000 mon.smithi097 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.97:0/1151410478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-11-17T06:52:11.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:11 smithi097 bash[24958]: audit 2023-11-17T06:52:10.675731+0000 mon.smithi097 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.97:0/1151410478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2023-11-17T06:52:12.094 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-17T06:52:12.094 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-17T06:52:12.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-17T06:52:12.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:12 smithi097 bash[24958]: cluster 2023-11-17T06:52:11.335391+0000 mgr.smithi097.swvonl (mgr.14180) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:12.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:12 smithi193 bash[28559]: cluster 2023-11-17T06:52:11.335391+0000 mgr.smithi097.swvonl (mgr.14180) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:14 smithi097 bash[24958]: cluster 2023-11-17T06:52:13.336031+0000 mgr.smithi097.swvonl (mgr.14180) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:14.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:14 smithi193 bash[28559]: cluster 2023-11-17T06:52:13.336031+0000 mgr.smithi097.swvonl (mgr.14180) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:16.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:16 smithi097 bash[24958]: cluster 2023-11-17T06:52:15.336704+0000 mgr.smithi097.swvonl (mgr.14180) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:16.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:16 smithi193 bash[28559]: cluster 2023-11-17T06:52:15.336704+0000 mgr.smithi097.swvonl (mgr.14180) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:17.228 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:52:18.676 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:18 smithi097 bash[24958]: cluster 2023-11-17T06:52:17.337267+0000 mgr.smithi097.swvonl (mgr.14180) 155 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:18 smithi193 bash[28559]: cluster 2023-11-17T06:52:17.337267+0000 mgr.smithi097.swvonl (mgr.14180) 155 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:20.680 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:20 smithi193 bash[28559]: cluster 2023-11-17T06:52:19.337732+0000 mgr.smithi097.swvonl (mgr.14180) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:20.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:20 smithi097 bash[24958]: cluster 2023-11-17T06:52:19.337732+0000 mgr.smithi097.swvonl (mgr.14180) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:21.456 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:52:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:22 smithi097 bash[24958]: cluster 2023-11-17T06:52:21.338331+0000 mgr.smithi097.swvonl (mgr.14180) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:22 smithi097 bash[24958]: audit 2023-11-17T06:52:21.445914+0000 mon.smithi097 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.97:0/4093710567' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-17T06:52:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:22 smithi193 bash[28559]: cluster 2023-11-17T06:52:21.338331+0000 mgr.smithi097.swvonl (mgr.14180) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:22 smithi193 bash[28559]: audit 2023-11-17T06:52:21.445914+0000 mon.smithi097 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.97:0/4093710567' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-17T06:52:23.088 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":18,"active_gid":14180,"active_name":"smithi097.swvonl","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":672483508},{"type":"v1","addr":"172.21.15.97:6801","nonce":672483508}]},"active_addr":"172.21.15.97:6801/672483508","active_change":"2023-11-17T06:48:41.280707+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":14196,"name":"smithi193.zqrhit","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.97:8443/","prometheus":"http://172.21.15.97:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":4,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":895467867}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":4046703312}]}]} 2023-11-17T06:52:23.091 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-17T06:52:23.091 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-17T06:52:23.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T06:52:24.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:24 smithi097 bash[24958]: cluster 2023-11-17T06:52:23.338860+0000 mgr.smithi097.swvonl (mgr.14180) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:24.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:24 smithi193 bash[28559]: cluster 2023-11-17T06:52:23.338860+0000 mgr.smithi097.swvonl (mgr.14180) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:26.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:26 smithi097 bash[24958]: cluster 2023-11-17T06:52:25.339428+0000 mgr.smithi097.swvonl (mgr.14180) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:26.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:26 smithi193 bash[28559]: cluster 2023-11-17T06:52:25.339428+0000 mgr.smithi097.swvonl (mgr.14180) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:27.985 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:52:28.760 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:28 smithi193 bash[28559]: cluster 2023-11-17T06:52:27.340003+0000 mgr.smithi097.swvonl (mgr.14180) 160 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:28.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:28 smithi097 bash[24958]: cluster 2023-11-17T06:52:27.340003+0000 mgr.smithi097.swvonl (mgr.14180) 160 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:30 smithi097 bash[24958]: cluster 2023-11-17T06:52:29.340600+0000 mgr.smithi097.swvonl (mgr.14180) 161 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:30 smithi097 bash[24958]: audit 2023-11-17T06:52:30.383075+0000 mon.smithi097 (mon.0) 581 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:30 smithi193 bash[28559]: cluster 2023-11-17T06:52:29.340600+0000 mgr.smithi097.swvonl (mgr.14180) 161 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:30 smithi193 bash[28559]: audit 2023-11-17T06:52:30.383075+0000 mon.smithi097 (mon.0) 581 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:32.274 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:52:32.276 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":41,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","created":"2023-11-17T06:46:29.288434+0000","modified":"2023-11-17T06:51:39.172152+0000","last_up_change":"2023-11-17T06:51:37.154487+0000","last_in_change":"2023-11-17T06:50:34.925718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T06:51:11.076187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8e0cefb6-3c13-4136-bee5-9a4a3589ccd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6801","nonce":2497341844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6803","nonce":2497341844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6807","nonce":2497341844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6805","nonce":2497341844}]},"public_addr":"172.21.15.193:6801/2497341844","cluster_addr":"172.21.15.193:6803/2497341844","heartbeat_back_addr":"172.21.15.193:6807/2497341844","heartbeat_front_addr":"172.21.15.193:6805/2497341844","state":["exists","up"]},{"osd":1,"uuid":"aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6803","nonce":3352242368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6805","nonce":3352242368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6809","nonce":3352242368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6807","nonce":3352242368}]},"public_addr":"172.21.15.97:6803/3352242368","cluster_addr":"172.21.15.97:6805/3352242368","heartbeat_back_addr":"172.21.15.97:6809/3352242368","heartbeat_front_addr":"172.21.15.97:6807/3352242368","state":["exists","up"]},{"osd":2,"uuid":"76c1caa8-14d7-43e5-bedf-44135cb7b231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6809","nonce":3825756302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6811","nonce":3825756302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6815","nonce":3825756302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6813","nonce":3825756302}]},"public_addr":"172.21.15.193:6809/3825756302","cluster_addr":"172.21.15.193:6811/3825756302","heartbeat_back_addr":"172.21.15.193:6815/3825756302","heartbeat_front_addr":"172.21.15.193:6813/3825756302","state":["exists","up"]},{"osd":3,"uuid":"9bb2d027-be64-4f25-92eb-0d72ee3f968c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6811","nonce":1152479819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6813","nonce":1152479819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6817","nonce":1152479819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6815","nonce":1152479819}]},"public_addr":"172.21.15.97:6811/1152479819","cluster_addr":"172.21.15.97:6813/1152479819","heartbeat_back_addr":"172.21.15.97:6817/1152479819","heartbeat_front_addr":"172.21.15.97:6815/1152479819","state":["exists","up"]},{"osd":4,"uuid":"8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6817","nonce":2167424517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6819","nonce":2167424517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6823","nonce":2167424517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6821","nonce":2167424517}]},"public_addr":"172.21.15.193:6817/2167424517","cluster_addr":"172.21.15.193:6819/2167424517","heartbeat_back_addr":"172.21.15.193:6823/2167424517","heartbeat_front_addr":"172.21.15.193:6821/2167424517","state":["exists","up"]},{"osd":5,"uuid":"7463f212-d4f5-427f-a4db-3835fce07e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6819","nonce":3902830634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6821","nonce":3902830634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6825","nonce":3902830634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6823","nonce":3902830634}]},"public_addr":"172.21.15.97:6819/3902830634","cluster_addr":"172.21.15.97:6821/3902830634","heartbeat_back_addr":"172.21.15.97:6825/3902830634","heartbeat_front_addr":"172.21.15.97:6823/3902830634","state":["exists","up"]},{"osd":6,"uuid":"2e7230c8-6a76-4ced-a43f-63071d3dde82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6825","nonce":2704435074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6827","nonce":2704435074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6831","nonce":2704435074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6829","nonce":2704435074}]},"public_addr":"172.21.15.193:6825/2704435074","cluster_addr":"172.21.15.193:6827/2704435074","heartbeat_back_addr":"172.21.15.193:6831/2704435074","heartbeat_front_addr":"172.21.15.193:6829/2704435074","state":["exists","up"]},{"osd":7,"uuid":"c4c1b9cb-3792-496d-b445-63a1e16c3823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6827","nonce":2156009798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6829","nonce":2156009798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6833","nonce":2156009798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6831","nonce":2156009798}]},"public_addr":"172.21.15.97:6827/2156009798","cluster_addr":"172.21.15.97:6829/2156009798","heartbeat_back_addr":"172.21.15.97:6833/2156009798","heartbeat_front_addr":"172.21.15.97:6831/2156009798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:01.239713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:09.387274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:06.883757+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:18.379782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:12.775627+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:27.442889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:17.833444+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:35.545858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/3905637356":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6800/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:0/1794480283":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/2127305343":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/404773595":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:6800/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/974873253":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6800/3584977603":"2023-11-18T06:46:57.468702+0000","172.21.15.97:0/396146213":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6801/3584977603":"2023-11-18T06:46:57.468702+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T06:52:32.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:32 smithi097 bash[24958]: cluster 2023-11-17T06:52:31.341228+0000 mgr.smithi097.swvonl (mgr.14180) 162 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:32.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:32 smithi193 bash[28559]: cluster 2023-11-17T06:52:31.341228+0000 mgr.smithi097.swvonl (mgr.14180) 162 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:33.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:33 smithi097 bash[24958]: audit 2023-11-17T06:52:32.271304+0000 mon.smithi097 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.97:0/814135144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:33.478 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-17T06:52:33.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd dump --format=json 2023-11-17T06:52:33.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:33 smithi193 bash[28559]: audit 2023-11-17T06:52:32.271304+0000 mon.smithi097 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.97:0/814135144' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:34.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:34 smithi097 bash[24958]: cluster 2023-11-17T06:52:33.341797+0000 mgr.smithi097.swvonl (mgr.14180) 163 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:34 smithi193 bash[28559]: cluster 2023-11-17T06:52:33.341797+0000 mgr.smithi097.swvonl (mgr.14180) 163 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:36 smithi097 bash[24958]: cluster 2023-11-17T06:52:35.342350+0000 mgr.smithi097.swvonl (mgr.14180) 164 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:36 smithi097 bash[24958]: audit 2023-11-17T06:52:36.242440+0000 mon.smithi097 (mon.0) 583 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:36 smithi193 bash[28559]: cluster 2023-11-17T06:52:35.342350+0000 mgr.smithi097.swvonl (mgr.14180) 164 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:36 smithi193 bash[28559]: audit 2023-11-17T06:52:36.242440+0000 mon.smithi097 (mon.0) 583 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:52:38.754 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:52:38.781 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:38 smithi097 bash[24958]: cluster 2023-11-17T06:52:37.342874+0000 mgr.smithi097.swvonl (mgr.14180) 165 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:38.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:38 smithi193 bash[28559]: cluster 2023-11-17T06:52:37.342874+0000 mgr.smithi097.swvonl (mgr.14180) 165 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:40.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:40 smithi097 bash[24958]: cluster 2023-11-17T06:52:39.343478+0000 mgr.smithi097.swvonl (mgr.14180) 166 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:40.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:40 smithi193 bash[28559]: cluster 2023-11-17T06:52:39.343478+0000 mgr.smithi097.swvonl (mgr.14180) 166 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:42.270 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:52:42.270 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":41,"fsid":"e74de5f6-8514-11ee-8dbf-212e2dc638e7","created":"2023-11-17T06:46:29.288434+0000","modified":"2023-11-17T06:51:39.172152+0000","last_up_change":"2023-11-17T06:51:37.154487+0000","last_in_change":"2023-11-17T06:50:34.925718+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-17T06:51:11.076187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"8e0cefb6-3c13-4136-bee5-9a4a3589ccd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6801","nonce":2497341844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6803","nonce":2497341844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6807","nonce":2497341844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2497341844},{"type":"v1","addr":"172.21.15.193:6805","nonce":2497341844}]},"public_addr":"172.21.15.193:6801/2497341844","cluster_addr":"172.21.15.193:6803/2497341844","heartbeat_back_addr":"172.21.15.193:6807/2497341844","heartbeat_front_addr":"172.21.15.193:6805/2497341844","state":["exists","up"]},{"osd":1,"uuid":"aaaf0bb5-ecd5-44cf-9414-460cad7f2cbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6803","nonce":3352242368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6805","nonce":3352242368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6809","nonce":3352242368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":3352242368},{"type":"v1","addr":"172.21.15.97:6807","nonce":3352242368}]},"public_addr":"172.21.15.97:6803/3352242368","cluster_addr":"172.21.15.97:6805/3352242368","heartbeat_back_addr":"172.21.15.97:6809/3352242368","heartbeat_front_addr":"172.21.15.97:6807/3352242368","state":["exists","up"]},{"osd":2,"uuid":"76c1caa8-14d7-43e5-bedf-44135cb7b231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6809","nonce":3825756302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6811","nonce":3825756302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6815","nonce":3825756302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3825756302},{"type":"v1","addr":"172.21.15.193:6813","nonce":3825756302}]},"public_addr":"172.21.15.193:6809/3825756302","cluster_addr":"172.21.15.193:6811/3825756302","heartbeat_back_addr":"172.21.15.193:6815/3825756302","heartbeat_front_addr":"172.21.15.193:6813/3825756302","state":["exists","up"]},{"osd":3,"uuid":"9bb2d027-be64-4f25-92eb-0d72ee3f968c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6811","nonce":1152479819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6813","nonce":1152479819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6817","nonce":1152479819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1152479819},{"type":"v1","addr":"172.21.15.97:6815","nonce":1152479819}]},"public_addr":"172.21.15.97:6811/1152479819","cluster_addr":"172.21.15.97:6813/1152479819","heartbeat_back_addr":"172.21.15.97:6817/1152479819","heartbeat_front_addr":"172.21.15.97:6815/1152479819","state":["exists","up"]},{"osd":4,"uuid":"8d647aa7-1ecb-40e6-b6ec-dd5feb38b2b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6817","nonce":2167424517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6819","nonce":2167424517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6823","nonce":2167424517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":2167424517},{"type":"v1","addr":"172.21.15.193:6821","nonce":2167424517}]},"public_addr":"172.21.15.193:6817/2167424517","cluster_addr":"172.21.15.193:6819/2167424517","heartbeat_back_addr":"172.21.15.193:6823/2167424517","heartbeat_front_addr":"172.21.15.193:6821/2167424517","state":["exists","up"]},{"osd":5,"uuid":"7463f212-d4f5-427f-a4db-3835fce07e46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6819","nonce":3902830634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6821","nonce":3902830634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6825","nonce":3902830634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":3902830634},{"type":"v1","addr":"172.21.15.97:6823","nonce":3902830634}]},"public_addr":"172.21.15.97:6819/3902830634","cluster_addr":"172.21.15.97:6821/3902830634","heartbeat_back_addr":"172.21.15.97:6825/3902830634","heartbeat_front_addr":"172.21.15.97:6823/3902830634","state":["exists","up"]},{"osd":6,"uuid":"2e7230c8-6a76-4ced-a43f-63071d3dde82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6825","nonce":2704435074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6827","nonce":2704435074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6831","nonce":2704435074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2704435074},{"type":"v1","addr":"172.21.15.193:6829","nonce":2704435074}]},"public_addr":"172.21.15.193:6825/2704435074","cluster_addr":"172.21.15.193:6827/2704435074","heartbeat_back_addr":"172.21.15.193:6831/2704435074","heartbeat_front_addr":"172.21.15.193:6829/2704435074","state":["exists","up"]},{"osd":7,"uuid":"c4c1b9cb-3792-496d-b445-63a1e16c3823","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6827","nonce":2156009798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6829","nonce":2156009798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6833","nonce":2156009798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":2156009798},{"type":"v1","addr":"172.21.15.97:6831","nonce":2156009798}]},"public_addr":"172.21.15.97:6827/2156009798","cluster_addr":"172.21.15.97:6829/2156009798","heartbeat_back_addr":"172.21.15.97:6833/2156009798","heartbeat_front_addr":"172.21.15.97:6831/2156009798","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:01.239713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:09.387274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:06.883757+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:18.379782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:12.775627+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:27.442889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:17.833444+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-17T06:51:35.545858+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/3905637356":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6800/1471680050":"2023-11-18T06:48:41.280362+0000","172.21.15.97:0/1794480283":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/2127305343":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/404773595":"2023-11-18T06:48:41.280362+0000","172.21.15.97:6801/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:6800/3548435767":"2023-11-18T06:47:39.524664+0000","172.21.15.97:0/974873253":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6800/3584977603":"2023-11-18T06:46:57.468702+0000","172.21.15.97:0/396146213":"2023-11-18T06:46:57.468702+0000","172.21.15.97:6801/3584977603":"2023-11-18T06:46:57.468702+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-17T06:52:42.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:42 smithi097 bash[24958]: cluster 2023-11-17T06:52:41.344030+0000 mgr.smithi097.swvonl (mgr.14180) 167 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:42.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:42 smithi193 bash[28559]: cluster 2023-11-17T06:52:41.344030+0000 mgr.smithi097.swvonl (mgr.14180) 167 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:43.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-17T06:52:43.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:43 smithi097 bash[24958]: audit 2023-11-17T06:52:42.266900+0000 mon.smithi097 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.97:0/310948072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:43.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:43 smithi193 bash[28559]: audit 2023-11-17T06:52:42.266900+0000 mon.smithi097 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.97:0/310948072' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-17T06:52:44.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:44 smithi097 bash[24958]: cluster 2023-11-17T06:52:43.344618+0000 mgr.smithi097.swvonl (mgr.14180) 168 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:44.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:44 smithi193 bash[28559]: cluster 2023-11-17T06:52:43.344618+0000 mgr.smithi097.swvonl (mgr.14180) 168 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:52:46.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:46 smithi097 bash[24958]: cluster 2023-11-17T06:52:45.345212+0000 mgr.smithi097.swvonl (mgr.14180) 169 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.508 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:17.565 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:48 smithi097 bash[24958]: cluster 2023-11-17T06:52:47.345720+0000 mgr.smithi097.swvonl (mgr.14180) 170 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.565 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:50 smithi097 bash[24958]: cluster 2023-11-17T06:52:49.346350+0000 mgr.smithi097.swvonl (mgr.14180) 171 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.567 INFO:teuthology.orchestra.run.smithi097.stdout:68719476761 2023-11-17T06:53:17.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-17T06:53:17.569 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:46 smithi193 bash[28559]: cluster 2023-11-17T06:52:45.345212+0000 mgr.smithi097.swvonl (mgr.14180) 169 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:48 smithi193 bash[28559]: cluster 2023-11-17T06:52:47.345720+0000 mgr.smithi097.swvonl (mgr.14180) 170 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:50 smithi193 bash[28559]: cluster 2023-11-17T06:52:49.346350+0000 mgr.smithi097.swvonl (mgr.14180) 171 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:52 smithi193 bash[28559]: cluster 2023-11-17T06:52:51.346933+0000 mgr.smithi097.swvonl (mgr.14180) 172 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:54 smithi193 bash[28559]: cluster 2023-11-17T06:52:53.347623+0000 mgr.smithi097.swvonl (mgr.14180) 173 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:56 smithi193 bash[28559]: cluster 2023-11-17T06:52:55.348260+0000 mgr.smithi097.swvonl (mgr.14180) 174 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:52:58 smithi193 bash[28559]: cluster 2023-11-17T06:52:57.348906+0000 mgr.smithi097.swvonl (mgr.14180) 175 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:00 smithi193 bash[28559]: cluster 2023-11-17T06:52:59.349521+0000 mgr.smithi097.swvonl (mgr.14180) 176 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.570 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:02 smithi193 bash[28559]: cluster 2023-11-17T06:53:01.350088+0000 mgr.smithi097.swvonl (mgr.14180) 177 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:04 smithi193 bash[28559]: cluster 2023-11-17T06:53:03.350768+0000 mgr.smithi097.swvonl (mgr.14180) 178 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:06 smithi193 bash[28559]: cluster 2023-11-17T06:53:05.351383+0000 mgr.smithi097.swvonl (mgr.14180) 179 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:08 smithi193 bash[28559]: cluster 2023-11-17T06:53:07.351936+0000 mgr.smithi097.swvonl (mgr.14180) 180 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:10 smithi193 bash[28559]: cluster 2023-11-17T06:53:09.352514+0000 mgr.smithi097.swvonl (mgr.14180) 181 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:12 smithi193 bash[28559]: cluster 2023-11-17T06:53:11.353071+0000 mgr.smithi097.swvonl (mgr.14180) 182 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:14 smithi193 bash[28559]: cluster 2023-11-17T06:53:13.353605+0000 mgr.smithi097.swvonl (mgr.14180) 183 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.571 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:16 smithi193 bash[28559]: cluster 2023-11-17T06:53:15.354197+0000 mgr.smithi097.swvonl (mgr.14180) 184 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:52 smithi097 bash[24958]: cluster 2023-11-17T06:52:51.346933+0000 mgr.smithi097.swvonl (mgr.14180) 172 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:54 smithi097 bash[24958]: cluster 2023-11-17T06:52:53.347623+0000 mgr.smithi097.swvonl (mgr.14180) 173 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:56 smithi097 bash[24958]: cluster 2023-11-17T06:52:55.348260+0000 mgr.smithi097.swvonl (mgr.14180) 174 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:52:58 smithi097 bash[24958]: cluster 2023-11-17T06:52:57.348906+0000 mgr.smithi097.swvonl (mgr.14180) 175 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:00 smithi097 bash[24958]: cluster 2023-11-17T06:52:59.349521+0000 mgr.smithi097.swvonl (mgr.14180) 176 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:02 smithi097 bash[24958]: cluster 2023-11-17T06:53:01.350088+0000 mgr.smithi097.swvonl (mgr.14180) 177 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:04 smithi097 bash[24958]: cluster 2023-11-17T06:53:03.350768+0000 mgr.smithi097.swvonl (mgr.14180) 178 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:06 smithi097 bash[24958]: cluster 2023-11-17T06:53:05.351383+0000 mgr.smithi097.swvonl (mgr.14180) 179 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:08 smithi097 bash[24958]: cluster 2023-11-17T06:53:07.351936+0000 mgr.smithi097.swvonl (mgr.14180) 180 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:10 smithi097 bash[24958]: cluster 2023-11-17T06:53:09.352514+0000 mgr.smithi097.swvonl (mgr.14180) 181 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:12 smithi097 bash[24958]: cluster 2023-11-17T06:53:11.353071+0000 mgr.smithi097.swvonl (mgr.14180) 182 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:14 smithi097 bash[24958]: cluster 2023-11-17T06:53:13.353605+0000 mgr.smithi097.swvonl (mgr.14180) 183 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:17.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:16 smithi097 bash[24958]: cluster 2023-11-17T06:53:15.354197+0000 mgr.smithi097.swvonl (mgr.14180) 184 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:18.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:18 smithi097 bash[24958]: cluster 2023-11-17T06:53:17.354771+0000 mgr.smithi097.swvonl (mgr.14180) 185 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:18 smithi193 bash[28559]: cluster 2023-11-17T06:53:17.354771+0000 mgr.smithi097.swvonl (mgr.14180) 185 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:20.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:20 smithi097 bash[24958]: cluster 2023-11-17T06:53:19.355381+0000 mgr.smithi097.swvonl (mgr.14180) 186 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:20.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:20 smithi193 bash[28559]: cluster 2023-11-17T06:53:19.355381+0000 mgr.smithi097.swvonl (mgr.14180) 186 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:22.379 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:22.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:22 smithi097 bash[24958]: cluster 2023-11-17T06:53:21.355969+0000 mgr.smithi097.swvonl (mgr.14180) 187 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:22 smithi193 bash[28559]: cluster 2023-11-17T06:53:21.355969+0000 mgr.smithi097.swvonl (mgr.14180) 187 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:24.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:24 smithi097 bash[24958]: cluster 2023-11-17T06:53:23.356434+0000 mgr.smithi097.swvonl (mgr.14180) 188 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:24.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:24 smithi193 bash[28559]: cluster 2023-11-17T06:53:23.356434+0000 mgr.smithi097.swvonl (mgr.14180) 188 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:26.539 INFO:teuthology.orchestra.run.smithi097.stdout:94489280542 2023-11-17T06:53:26.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-17T06:53:26.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:26 smithi097 bash[24958]: cluster 2023-11-17T06:53:25.356914+0000 mgr.smithi097.swvonl (mgr.14180) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:26.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:26 smithi193 bash[28559]: cluster 2023-11-17T06:53:25.356914+0000 mgr.smithi097.swvonl (mgr.14180) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:28.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:28 smithi097 bash[24958]: cluster 2023-11-17T06:53:27.357451+0000 mgr.smithi097.swvonl (mgr.14180) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:28.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:28 smithi193 bash[28559]: cluster 2023-11-17T06:53:27.357451+0000 mgr.smithi097.swvonl (mgr.14180) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:30 smithi097 bash[24958]: cluster 2023-11-17T06:53:29.358064+0000 mgr.smithi097.swvonl (mgr.14180) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:30 smithi193 bash[28559]: cluster 2023-11-17T06:53:29.358064+0000 mgr.smithi097.swvonl (mgr.14180) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:31.322 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:32.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:32 smithi097 bash[24958]: cluster 2023-11-17T06:53:31.358666+0000 mgr.smithi097.swvonl (mgr.14180) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:32.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:32 smithi193 bash[28559]: cluster 2023-11-17T06:53:31.358666+0000 mgr.smithi097.swvonl (mgr.14180) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:34.497 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:34 smithi097 bash[24958]: cluster 2023-11-17T06:53:33.359280+0000 mgr.smithi097.swvonl (mgr.14180) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:34.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:34 smithi193 bash[28559]: cluster 2023-11-17T06:53:33.359280+0000 mgr.smithi097.swvonl (mgr.14180) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:35.320 INFO:teuthology.orchestra.run.smithi097.stdout:85899345952 2023-11-17T06:53:35.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-17T06:53:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:36 smithi097 bash[24958]: cluster 2023-11-17T06:53:35.359873+0000 mgr.smithi097.swvonl (mgr.14180) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:36 smithi097 bash[24958]: audit 2023-11-17T06:53:36.248524+0000 mon.smithi097 (mon.0) 585 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:53:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:36 smithi193 bash[28559]: cluster 2023-11-17T06:53:35.359873+0000 mgr.smithi097.swvonl (mgr.14180) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:36 smithi193 bash[28559]: audit 2023-11-17T06:53:36.248524+0000 mon.smithi097 (mon.0) 585 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:53:38.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:38 smithi097 bash[24958]: cluster 2023-11-17T06:53:37.360433+0000 mgr.smithi097.swvonl (mgr.14180) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:38.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:38 smithi193 bash[28559]: cluster 2023-11-17T06:53:37.360433+0000 mgr.smithi097.swvonl (mgr.14180) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:40.103 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:40.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:40 smithi097 bash[24958]: cluster 2023-11-17T06:53:39.361001+0000 mgr.smithi097.swvonl (mgr.14180) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:40.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:40 smithi193 bash[28559]: cluster 2023-11-17T06:53:39.361001+0000 mgr.smithi097.swvonl (mgr.14180) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:42.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:41.336411+0000 mon.smithi097 (mon.0) 586 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: cluster 2023-11-17T06:53:41.361517+0000 mgr.smithi097.swvonl (mgr.14180) 197 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:41.770971+0000 mon.smithi097 (mon.0) 587 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:42.061498+0000 mon.smithi097 (mon.0) 588 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:42.063184+0000 mon.smithi097 (mon.0) 589 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:42.070762+0000 mon.smithi097 (mon.0) 590 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:42 smithi097 bash[24958]: audit 2023-11-17T06:53:42.076455+0000 mon.smithi097 (mon.0) 591 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:53:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:41.336411+0000 mon.smithi097 (mon.0) 586 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: cluster 2023-11-17T06:53:41.361517+0000 mgr.smithi097.swvonl (mgr.14180) 197 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:41.770971+0000 mon.smithi097 (mon.0) 587 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:42.061498+0000 mon.smithi097 (mon.0) 588 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:42.063184+0000 mon.smithi097 (mon.0) 589 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:42.070762+0000 mon.smithi097 (mon.0) 590 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:53:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:42 smithi193 bash[28559]: audit 2023-11-17T06:53:42.076455+0000 mon.smithi097 (mon.0) 591 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:53:44.358 INFO:teuthology.orchestra.run.smithi097.stdout:128849018910 2023-11-17T06:53:44.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-17T06:53:44.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:44 smithi097 bash[24958]: cluster 2023-11-17T06:53:43.362036+0000 mgr.smithi097.swvonl (mgr.14180) 198 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:44.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:44 smithi193 bash[28559]: cluster 2023-11-17T06:53:43.362036+0000 mgr.smithi097.swvonl (mgr.14180) 198 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:46.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:46 smithi097 bash[24958]: cluster 2023-11-17T06:53:45.362718+0000 mgr.smithi097.swvonl (mgr.14180) 199 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:46.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:46 smithi193 bash[28559]: cluster 2023-11-17T06:53:45.362718+0000 mgr.smithi097.swvonl (mgr.14180) 199 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:48 smithi097 bash[24958]: cluster 2023-11-17T06:53:47.363339+0000 mgr.smithi097.swvonl (mgr.14180) 200 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:48.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:48 smithi193 bash[28559]: cluster 2023-11-17T06:53:47.363339+0000 mgr.smithi097.swvonl (mgr.14180) 200 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:49.143 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:50.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:50 smithi097 bash[24958]: cluster 2023-11-17T06:53:49.364004+0000 mgr.smithi097.swvonl (mgr.14180) 201 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:50.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:50 smithi193 bash[28559]: cluster 2023-11-17T06:53:49.364004+0000 mgr.smithi097.swvonl (mgr.14180) 201 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:52.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:52 smithi097 bash[24958]: cluster 2023-11-17T06:53:51.364710+0000 mgr.smithi097.swvonl (mgr.14180) 202 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:52 smithi193 bash[28559]: cluster 2023-11-17T06:53:51.364710+0000 mgr.smithi097.swvonl (mgr.14180) 202 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:53.208 INFO:teuthology.orchestra.run.smithi097.stdout:111669149729 2023-11-17T06:53:53.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-17T06:53:54.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:54 smithi097 bash[24958]: cluster 2023-11-17T06:53:53.365205+0000 mgr.smithi097.swvonl (mgr.14180) 203 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:54.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:54 smithi193 bash[28559]: cluster 2023-11-17T06:53:53.365205+0000 mgr.smithi097.swvonl (mgr.14180) 203 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:56 smithi097 bash[24958]: cluster 2023-11-17T06:53:55.365757+0000 mgr.smithi097.swvonl (mgr.14180) 204 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:56.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:56 smithi193 bash[28559]: cluster 2023-11-17T06:53:55.365757+0000 mgr.smithi097.swvonl (mgr.14180) 204 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:57.995 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:53:58.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:53:58 smithi097 bash[24958]: cluster 2023-11-17T06:53:57.366344+0000 mgr.smithi097.swvonl (mgr.14180) 205 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:53:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:53:58 smithi193 bash[28559]: cluster 2023-11-17T06:53:57.366344+0000 mgr.smithi097.swvonl (mgr.14180) 205 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:00.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:00 smithi097 bash[24958]: cluster 2023-11-17T06:53:59.366810+0000 mgr.smithi097.swvonl (mgr.14180) 206 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:00.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:00 smithi193 bash[28559]: cluster 2023-11-17T06:53:59.366810+0000 mgr.smithi097.swvonl (mgr.14180) 206 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:02.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:02 smithi097 bash[24958]: cluster 2023-11-17T06:54:01.367304+0000 mgr.smithi097.swvonl (mgr.14180) 207 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:02.406 INFO:teuthology.orchestra.run.smithi097.stdout:150323855394 2023-11-17T06:54:02.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-17T06:54:02.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:02 smithi193 bash[28559]: cluster 2023-11-17T06:54:01.367304+0000 mgr.smithi097.swvonl (mgr.14180) 207 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:04.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:04 smithi097 bash[24958]: cluster 2023-11-17T06:54:03.368007+0000 mgr.smithi097.swvonl (mgr.14180) 208 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:04.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:04 smithi193 bash[28559]: cluster 2023-11-17T06:54:03.368007+0000 mgr.smithi097.swvonl (mgr.14180) 208 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:06.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:06 smithi097 bash[24958]: cluster 2023-11-17T06:54:05.368647+0000 mgr.smithi097.swvonl (mgr.14180) 209 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:06.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:06 smithi193 bash[28559]: cluster 2023-11-17T06:54:05.368647+0000 mgr.smithi097.swvonl (mgr.14180) 209 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:07.194 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:08 smithi097 bash[24958]: cluster 2023-11-17T06:54:07.369213+0000 mgr.smithi097.swvonl (mgr.14180) 210 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:08.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:08 smithi193 bash[28559]: cluster 2023-11-17T06:54:07.369213+0000 mgr.smithi097.swvonl (mgr.14180) 210 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:10.712 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:10 smithi097 bash[24958]: cluster 2023-11-17T06:54:09.369774+0000 mgr.smithi097.swvonl (mgr.14180) 211 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:10 smithi193 bash[28559]: cluster 2023-11-17T06:54:09.369774+0000 mgr.smithi097.swvonl (mgr.14180) 211 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:11.340 INFO:teuthology.orchestra.run.smithi097.stdout:128849018917 2023-11-17T06:54:11.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-17T06:54:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:12 smithi097 bash[24958]: cluster 2023-11-17T06:54:11.370367+0000 mgr.smithi097.swvonl (mgr.14180) 212 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:12.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:12 smithi193 bash[28559]: cluster 2023-11-17T06:54:11.370367+0000 mgr.smithi097.swvonl (mgr.14180) 212 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:14 smithi097 bash[24958]: cluster 2023-11-17T06:54:13.370948+0000 mgr.smithi097.swvonl (mgr.14180) 213 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:14.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:14 smithi193 bash[28559]: cluster 2023-11-17T06:54:13.370948+0000 mgr.smithi097.swvonl (mgr.14180) 213 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:16.122 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:16.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:16 smithi097 bash[24958]: cluster 2023-11-17T06:54:15.371605+0000 mgr.smithi097.swvonl (mgr.14180) 214 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:16.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:16 smithi193 bash[28559]: cluster 2023-11-17T06:54:15.371605+0000 mgr.smithi097.swvonl (mgr.14180) 214 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:18 smithi097 bash[24958]: cluster 2023-11-17T06:54:17.372192+0000 mgr.smithi097.swvonl (mgr.14180) 215 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:18 smithi193 bash[28559]: cluster 2023-11-17T06:54:17.372192+0000 mgr.smithi097.swvonl (mgr.14180) 215 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:20.492 INFO:teuthology.orchestra.run.smithi097.stdout:167503724579 2023-11-17T06:54:20.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-17T06:54:20.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:20 smithi097 bash[24958]: cluster 2023-11-17T06:54:19.372812+0000 mgr.smithi097.swvonl (mgr.14180) 216 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:20.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:20 smithi193 bash[28559]: cluster 2023-11-17T06:54:19.372812+0000 mgr.smithi097.swvonl (mgr.14180) 216 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:22 smithi097 bash[24958]: cluster 2023-11-17T06:54:21.373552+0000 mgr.smithi097.swvonl (mgr.14180) 217 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:22.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:22 smithi193 bash[28559]: cluster 2023-11-17T06:54:21.373552+0000 mgr.smithi097.swvonl (mgr.14180) 217 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:24.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:24 smithi097 bash[24958]: cluster 2023-11-17T06:54:23.374352+0000 mgr.smithi097.swvonl (mgr.14180) 218 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:24.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:24 smithi193 bash[28559]: cluster 2023-11-17T06:54:23.374352+0000 mgr.smithi097.swvonl (mgr.14180) 218 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:26.742 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:26 smithi097 bash[24958]: cluster 2023-11-17T06:54:25.375061+0000 mgr.smithi097.swvonl (mgr.14180) 219 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:26.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:26 smithi193 bash[28559]: cluster 2023-11-17T06:54:25.375061+0000 mgr.smithi097.swvonl (mgr.14180) 219 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:28.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:28 smithi097 bash[24958]: cluster 2023-11-17T06:54:27.375693+0000 mgr.smithi097.swvonl (mgr.14180) 220 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:28.619 INFO:teuthology.orchestra.run.smithi097.stdout:68719476780 2023-11-17T06:54:28.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:28 smithi193 bash[28559]: cluster 2023-11-17T06:54:27.375693+0000 mgr.smithi097.swvonl (mgr.14180) 220 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:29.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 68719476761 got 68719476780 for osd.0 2023-11-17T06:54:29.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-17T06:54:29.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:29 smithi097 bash[24958]: audit 2023-11-17T06:54:28.616978+0000 mon.smithi097 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.97:0/843254442' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T06:54:29.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:29 smithi193 bash[28559]: audit 2023-11-17T06:54:28.616978+0000 mon.smithi097 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.97:0/843254442' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-17T06:54:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:30 smithi097 bash[24958]: cluster 2023-11-17T06:54:29.376191+0000 mgr.smithi097.swvonl (mgr.14180) 221 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:30 smithi193 bash[28559]: cluster 2023-11-17T06:54:29.376191+0000 mgr.smithi097.swvonl (mgr.14180) 221 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:32.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:32 smithi097 bash[24958]: cluster 2023-11-17T06:54:31.376727+0000 mgr.smithi097.swvonl (mgr.14180) 222 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:32.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:32 smithi193 bash[28559]: cluster 2023-11-17T06:54:31.376727+0000 mgr.smithi097.swvonl (mgr.14180) 222 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:34.269 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:34.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:34 smithi097 bash[24958]: cluster 2023-11-17T06:54:33.377312+0000 mgr.smithi097.swvonl (mgr.14180) 223 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:34.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:34 smithi193 bash[28559]: cluster 2023-11-17T06:54:33.377312+0000 mgr.smithi097.swvonl (mgr.14180) 223 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:36 smithi097 bash[24958]: cluster 2023-11-17T06:54:35.377839+0000 mgr.smithi097.swvonl (mgr.14180) 224 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:36 smithi193 bash[28559]: cluster 2023-11-17T06:54:35.377839+0000 mgr.smithi097.swvonl (mgr.14180) 224 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:37.761 INFO:teuthology.orchestra.run.smithi097.stdout:94489280557 2023-11-17T06:54:38.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 94489280542 got 94489280557 for osd.1 2023-11-17T06:54:38.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-17T06:54:38.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:38 smithi097 bash[24958]: cluster 2023-11-17T06:54:37.378456+0000 mgr.smithi097.swvonl (mgr.14180) 225 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:38.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:38 smithi097 bash[24958]: audit 2023-11-17T06:54:37.758781+0000 mon.smithi097 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.97:0/4147918687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T06:54:38.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:38 smithi193 bash[28559]: cluster 2023-11-17T06:54:37.378456+0000 mgr.smithi097.swvonl (mgr.14180) 225 : cluster [DBG] pgmap v208: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:38.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:38 smithi193 bash[28559]: audit 2023-11-17T06:54:37.758781+0000 mon.smithi097 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.97:0/4147918687' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-17T06:54:40.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:40 smithi097 bash[24958]: cluster 2023-11-17T06:54:39.379001+0000 mgr.smithi097.swvonl (mgr.14180) 226 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:40.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:40 smithi193 bash[28559]: cluster 2023-11-17T06:54:39.379001+0000 mgr.smithi097.swvonl (mgr.14180) 226 : cluster [DBG] pgmap v209: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:42 smithi097 bash[24958]: cluster 2023-11-17T06:54:41.379550+0000 mgr.smithi097.swvonl (mgr.14180) 227 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:42 smithi097 bash[24958]: audit 2023-11-17T06:54:42.087724+0000 mon.smithi097 (mon.0) 594 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:54:42.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:42 smithi193 bash[28559]: cluster 2023-11-17T06:54:41.379550+0000 mgr.smithi097.swvonl (mgr.14180) 227 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:42.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:42 smithi193 bash[28559]: audit 2023-11-17T06:54:42.087724+0000 mon.smithi097 (mon.0) 594 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:54:43.457 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:44 smithi097 bash[24958]: cluster 2023-11-17T06:54:43.380118+0000 mgr.smithi097.swvonl (mgr.14180) 228 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:44.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:44 smithi193 bash[28559]: cluster 2023-11-17T06:54:43.380118+0000 mgr.smithi097.swvonl (mgr.14180) 228 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:46.711 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:46 smithi097 bash[24958]: cluster 2023-11-17T06:54:45.380671+0000 mgr.smithi097.swvonl (mgr.14180) 229 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:46.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:46 smithi193 bash[28559]: cluster 2023-11-17T06:54:45.380671+0000 mgr.smithi097.swvonl (mgr.14180) 229 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:47.094 INFO:teuthology.orchestra.run.smithi097.stdout:85899345967 2023-11-17T06:54:47.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:47 smithi097 bash[24958]: audit 2023-11-17T06:54:47.092497+0000 mon.smithi097 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.97:0/1874873350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T06:54:47.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:47 smithi193 bash[28559]: audit 2023-11-17T06:54:47.092497+0000 mon.smithi097 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.97:0/1874873350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-17T06:54:48.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 85899345952 got 85899345967 for osd.2 2023-11-17T06:54:48.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: cluster 2023-11-17T06:54:47.381100+0000 mgr.smithi097.swvonl (mgr.14180) 230 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.636027+0000 mon.smithi097 (mon.0) 596 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.703366+0000 mon.smithi097 (mon.0) 597 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.965280+0000 mon.smithi097 (mon.0) 598 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.966475+0000 mon.smithi097 (mon.0) 599 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:54:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.974556+0000 mon.smithi097 (mon.0) 600 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:48 smithi097 bash[24958]: audit 2023-11-17T06:54:47.981917+0000 mon.smithi097 (mon.0) 601 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: cluster 2023-11-17T06:54:47.381100+0000 mgr.smithi097.swvonl (mgr.14180) 230 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.636027+0000 mon.smithi097 (mon.0) 596 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.703366+0000 mon.smithi097 (mon.0) 597 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.965280+0000 mon.smithi097 (mon.0) 598 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.966475+0000 mon.smithi097 (mon.0) 599 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.974556+0000 mon.smithi097 (mon.0) 600 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:54:49.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:48 smithi193 bash[28559]: audit 2023-11-17T06:54:47.981917+0000 mon.smithi097 (mon.0) 601 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:54:51.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:50 smithi097 bash[24958]: cluster 2023-11-17T06:54:49.381685+0000 mgr.smithi097.swvonl (mgr.14180) 231 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:51.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:50 smithi193 bash[28559]: cluster 2023-11-17T06:54:49.381685+0000 mgr.smithi097.swvonl (mgr.14180) 231 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:52.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:52 smithi097 bash[24958]: cluster 2023-11-17T06:54:51.382266+0000 mgr.smithi097.swvonl (mgr.14180) 232 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:52.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:52 smithi193 bash[28559]: cluster 2023-11-17T06:54:51.382266+0000 mgr.smithi097.swvonl (mgr.14180) 232 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:52.791 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:54:54.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:54 smithi097 bash[24958]: cluster 2023-11-17T06:54:53.382843+0000 mgr.smithi097.swvonl (mgr.14180) 233 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:54.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:54 smithi193 bash[28559]: cluster 2023-11-17T06:54:53.382843+0000 mgr.smithi097.swvonl (mgr.14180) 233 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:56.114 INFO:teuthology.orchestra.run.smithi097.stdout:128849018925 2023-11-17T06:54:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:56 smithi097 bash[24958]: cluster 2023-11-17T06:54:55.383454+0000 mgr.smithi097.swvonl (mgr.14180) 234 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:56 smithi097 bash[24958]: audit 2023-11-17T06:54:56.111530+0000 mon.smithi097 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.97:0/219753336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T06:54:56.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:56 smithi193 bash[28559]: cluster 2023-11-17T06:54:55.383454+0000 mgr.smithi097.swvonl (mgr.14180) 234 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:56.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:56 smithi193 bash[28559]: audit 2023-11-17T06:54:56.111530+0000 mon.smithi097 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.97:0/219753336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-17T06:54:56.896 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018925 for osd.3 2023-11-17T06:54:56.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-17T06:54:58.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:54:58 smithi097 bash[24958]: cluster 2023-11-17T06:54:57.384143+0000 mgr.smithi097.swvonl (mgr.14180) 235 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:54:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:54:58 smithi193 bash[28559]: cluster 2023-11-17T06:54:57.384143+0000 mgr.smithi097.swvonl (mgr.14180) 235 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:00.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:00 smithi097 bash[24958]: cluster 2023-11-17T06:54:59.384878+0000 mgr.smithi097.swvonl (mgr.14180) 236 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:00.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:00 smithi193 bash[28559]: cluster 2023-11-17T06:54:59.384878+0000 mgr.smithi097.swvonl (mgr.14180) 236 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:01.686 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:02.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:02 smithi097 bash[24958]: cluster 2023-11-17T06:55:01.385464+0000 mgr.smithi097.swvonl (mgr.14180) 237 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:02.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:02 smithi193 bash[28559]: cluster 2023-11-17T06:55:01.385464+0000 mgr.smithi097.swvonl (mgr.14180) 237 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:04.750 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:04 smithi097 bash[24958]: cluster 2023-11-17T06:55:03.386012+0000 mgr.smithi097.swvonl (mgr.14180) 238 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:04.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:04 smithi193 bash[28559]: cluster 2023-11-17T06:55:03.386012+0000 mgr.smithi097.swvonl (mgr.14180) 238 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:05.098 INFO:teuthology.orchestra.run.smithi097.stdout:111669149743 2023-11-17T06:55:05.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:05 smithi193 bash[28559]: audit 2023-11-17T06:55:05.095862+0000 mon.smithi097 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.97:0/323430249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T06:55:05.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:05 smithi097 bash[24958]: audit 2023-11-17T06:55:05.095862+0000 mon.smithi097 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.97:0/323430249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-17T06:55:05.931 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149743 for osd.4 2023-11-17T06:55:05.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-17T06:55:06.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:06 smithi097 bash[24958]: cluster 2023-11-17T06:55:05.386621+0000 mgr.smithi097.swvonl (mgr.14180) 239 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:06.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:06 smithi193 bash[28559]: cluster 2023-11-17T06:55:05.386621+0000 mgr.smithi097.swvonl (mgr.14180) 239 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:08.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:08 smithi097 bash[24958]: cluster 2023-11-17T06:55:07.387137+0000 mgr.smithi097.swvonl (mgr.14180) 240 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:08.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:08 smithi193 bash[28559]: cluster 2023-11-17T06:55:07.387137+0000 mgr.smithi097.swvonl (mgr.14180) 240 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:10.719 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:10.746 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:10 smithi097 bash[24958]: cluster 2023-11-17T06:55:09.387711+0000 mgr.smithi097.swvonl (mgr.14180) 241 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:10 smithi193 bash[28559]: cluster 2023-11-17T06:55:09.387711+0000 mgr.smithi097.swvonl (mgr.14180) 241 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:12.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:12 smithi097 bash[24958]: cluster 2023-11-17T06:55:11.388365+0000 mgr.smithi097.swvonl (mgr.14180) 242 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:12 smithi193 bash[28559]: cluster 2023-11-17T06:55:11.388365+0000 mgr.smithi097.swvonl (mgr.14180) 242 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:14.146 INFO:teuthology.orchestra.run.smithi097.stdout:150323855408 2023-11-17T06:55:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:14 smithi097 bash[24958]: cluster 2023-11-17T06:55:13.389003+0000 mgr.smithi097.swvonl (mgr.14180) 243 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:14 smithi097 bash[24958]: audit 2023-11-17T06:55:14.143480+0000 mon.smithi097 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.97:0/3289595956' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T06:55:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:14 smithi193 bash[28559]: cluster 2023-11-17T06:55:13.389003+0000 mgr.smithi097.swvonl (mgr.14180) 243 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:14 smithi193 bash[28559]: audit 2023-11-17T06:55:14.143480+0000 mon.smithi097 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.97:0/3289595956' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-17T06:55:14.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855394 got 150323855408 for osd.5 2023-11-17T06:55:14.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-17T06:55:16.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:16 smithi097 bash[24958]: cluster 2023-11-17T06:55:15.389590+0000 mgr.smithi097.swvonl (mgr.14180) 244 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:16.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:16 smithi193 bash[28559]: cluster 2023-11-17T06:55:15.389590+0000 mgr.smithi097.swvonl (mgr.14180) 244 : cluster [DBG] pgmap v227: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:18 smithi097 bash[24958]: cluster 2023-11-17T06:55:17.390159+0000 mgr.smithi097.swvonl (mgr.14180) 245 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:18 smithi193 bash[28559]: cluster 2023-11-17T06:55:17.390159+0000 mgr.smithi097.swvonl (mgr.14180) 245 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:19.784 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:20.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:20 smithi097 bash[24958]: cluster 2023-11-17T06:55:19.390749+0000 mgr.smithi097.swvonl (mgr.14180) 246 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:20 smithi193 bash[28559]: cluster 2023-11-17T06:55:19.390749+0000 mgr.smithi097.swvonl (mgr.14180) 246 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:22.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:22 smithi097 bash[24958]: cluster 2023-11-17T06:55:21.391347+0000 mgr.smithi097.swvonl (mgr.14180) 247 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:22.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:22 smithi193 bash[28559]: cluster 2023-11-17T06:55:21.391347+0000 mgr.smithi097.swvonl (mgr.14180) 247 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:23.248 INFO:teuthology.orchestra.run.smithi097.stdout:128849018931 2023-11-17T06:55:23.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:23 smithi097 bash[24958]: audit 2023-11-17T06:55:23.245377+0000 mon.smithi097 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.97:0/4174844020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T06:55:23.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:23 smithi193 bash[28559]: audit 2023-11-17T06:55:23.245377+0000 mon.smithi097 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.97:0/4174844020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-17T06:55:24.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018917 got 128849018931 for osd.6 2023-11-17T06:55:24.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-17T06:55:24.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:24 smithi097 bash[24958]: cluster 2023-11-17T06:55:23.391944+0000 mgr.smithi097.swvonl (mgr.14180) 248 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:24 smithi193 bash[28559]: cluster 2023-11-17T06:55:23.391944+0000 mgr.smithi097.swvonl (mgr.14180) 248 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:26.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:26 smithi097 bash[24958]: cluster 2023-11-17T06:55:25.392608+0000 mgr.smithi097.swvonl (mgr.14180) 249 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:26.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:26 smithi193 bash[28559]: cluster 2023-11-17T06:55:25.392608+0000 mgr.smithi097.swvonl (mgr.14180) 249 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:28.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:28 smithi097 bash[24958]: cluster 2023-11-17T06:55:27.393160+0000 mgr.smithi097.swvonl (mgr.14180) 250 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:28 smithi193 bash[28559]: cluster 2023-11-17T06:55:27.393160+0000 mgr.smithi097.swvonl (mgr.14180) 250 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:28.894 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:30.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:30 smithi097 bash[24958]: cluster 2023-11-17T06:55:29.393747+0000 mgr.smithi097.swvonl (mgr.14180) 251 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:30.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:30 smithi193 bash[28559]: cluster 2023-11-17T06:55:29.393747+0000 mgr.smithi097.swvonl (mgr.14180) 251 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:32.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:32 smithi097 bash[24958]: cluster 2023-11-17T06:55:31.394367+0000 mgr.smithi097.swvonl (mgr.14180) 252 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:32.461 INFO:teuthology.orchestra.run.smithi097.stdout:167503724593 2023-11-17T06:55:32.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:32 smithi193 bash[28559]: cluster 2023-11-17T06:55:31.394367+0000 mgr.smithi097.swvonl (mgr.14180) 252 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:33.289 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724579 got 167503724593 for osd.7 2023-11-17T06:55:33.289 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-17T06:55:33.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T06:55:33.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:33 smithi097 bash[24958]: audit 2023-11-17T06:55:32.458779+0000 mon.smithi097 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.97:0/498821929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T06:55:33.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:33 smithi193 bash[28559]: audit 2023-11-17T06:55:32.458779+0000 mon.smithi097 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.97:0/498821929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-17T06:55:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:34 smithi097 bash[24958]: cluster 2023-11-17T06:55:33.394830+0000 mgr.smithi097.swvonl (mgr.14180) 253 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:34 smithi193 bash[28559]: cluster 2023-11-17T06:55:33.394830+0000 mgr.smithi097.swvonl (mgr.14180) 253 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:36.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:36 smithi097 bash[24958]: cluster 2023-11-17T06:55:35.395386+0000 mgr.smithi097.swvonl (mgr.14180) 254 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:36.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:36 smithi193 bash[28559]: cluster 2023-11-17T06:55:35.395386+0000 mgr.smithi097.swvonl (mgr.14180) 254 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:38.071 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:38.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:38 smithi097 bash[24958]: cluster 2023-11-17T06:55:37.396010+0000 mgr.smithi097.swvonl (mgr.14180) 255 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:38.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:38 smithi193 bash[28559]: cluster 2023-11-17T06:55:37.396010+0000 mgr.smithi097.swvonl (mgr.14180) 255 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:40.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:40 smithi097 bash[24958]: cluster 2023-11-17T06:55:39.396613+0000 mgr.smithi097.swvonl (mgr.14180) 256 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:40.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:40 smithi193 bash[28559]: cluster 2023-11-17T06:55:39.396613+0000 mgr.smithi097.swvonl (mgr.14180) 256 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:41.586 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:55:41.589 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-11-17T06:55:42.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:42 smithi097 bash[24958]: cluster 2023-11-17T06:55:41.396950+0000 mgr.smithi097.swvonl (mgr.14180) 257 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:42.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:42 smithi097 bash[24958]: audit 2023-11-17T06:55:41.576938+0000 mgr.smithi097.swvonl (mgr.14180) 258 : audit [DBG] from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:55:42.442 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2023-11-17T06:55:41.396802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749928448,"kb_used":2379964,"kb_used_data":3196,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747548484,"statfs":{"total":767926730752,"available":765489647616,"internally_reserved":0,"allocated":3272704,"data_stored":1276661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-17T06:51:39.449751+0000","last_change":"2023-11-17T06:51:39.449751+0000","last_active":"2023-11-17T06:51:39.449751+0000","last_peered":"2023-11-17T06:51:39.449751+0000","last_clean":"2023-11-17T06:51:39.449751+0000","last_became_active":"2023-11-17T06:51:39.449303+0000","last_became_peered":"2023-11-17T06:51:39.449303+0000","last_unstale":"2023-11-17T06:51:39.449751+0000","last_undegraded":"2023-11-17T06:51:39.449751+0000","last_fullsized":"2023-11-17T06:51:39.449751+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-17T06:51:11.613640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-17T06:51:11.613640+0000","last_clean_scrub_stamp":"2023-11-17T06:51:11.613640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":39,"seq":167503724595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297492,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443564,"statfs":{"total":95990841344,"available":95686209536,"internally_reserved":0,"allocated":405504,"data_stored":159307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.84799999999999998,"15min":0.85699999999999998},"min":{"1min":0.64400000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.98999999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.627,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0940000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.876}]},{"osd":1,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.69599999999999995,"15min":0.67400000000000004},"min":{"1min":0.56499999999999995,"5min":0.312,"15min":0.312},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.73699999999999999,"15min":0.69599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0269999999999999,"5min":1.26,"15min":1.26},"last":1.036},{"interface":"front","average":{"1min":0.88,"5min":0.89900000000000002,"15min":0.89500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1160000000000001,"5min":1.226,"15min":1.226},"last":0.752}]},{"osd":3,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79000000000000004,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.85499999999999998}]},{"osd":4,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.92000000000000004,"15min":0.91500000000000004},"min":{"1min":0.70299999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.135,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0049999999999999,"15min":1.004},"min":{"1min":0.68200000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1539999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.0680000000000001}]},{"osd":5,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.99399999999999999},"min":{"1min":0.503,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.198,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.123},{"interface":"front","average":{"1min":1.05,"5min":1.0149999999999999,"15min":1.006},"min":{"1min":0.51400000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.141}]},{"osd":6,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.0980000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.83199999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5509999999999999,"5min":1.766,"15min":1.766},"last":1.242}]}]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":1.121},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.92800000000000005,"15min":0.91800000000000004},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":1.101}]},{"osd":1,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70999999999999996,"15min":0.68200000000000005},"min":{"1min":0.47899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.03,"5min":1.133,"15min":1.133},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.68000000000000005},"min":{"1min":0.495,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98099999999999998,"5min":1.212,"15min":1.212},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.90600000000000003,"15min":0.91600000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":1.002}]},{"osd":3,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.51800000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1599999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.57699999999999996,"5min":0.436,"15min":0.436},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":1.048}]},{"osd":4,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98299999999999998,"15min":0.98999999999999999},"min":{"1min":0.748,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.169,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.042,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.81499999999999995,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.2909999999999999,"5min":1.385,"15min":1.385},"last":1.1970000000000001}]},{"osd":6,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0049999999999999,"15min":1.01},"min":{"1min":0.67400000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.1950000000000001,"5min":1.276,"15min":1.276},"last":1.214},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.026,"15min":1.0349999999999999},"min":{"1min":0.80200000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.1459999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":1.1739999999999999}]},{"osd":7,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.98099999999999998,"15min":0.97299999999999998},"min":{"1min":0.68200000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0209999999999999,"15min":1.0149999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2609999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.153}]}]},{"osd":2,"up_from":20,"seq":85899345978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72299999999999998,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.216,"5min":1.276,"15min":1.276},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69399999999999995},"min":{"1min":0.59599999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.85799999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.70099999999999996}]},{"osd":1,"last update":"Fri Nov 17 06:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85299999999999998,"15min":0.83599999999999997},"min":{"1min":0.70499999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1160000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.91500000000000004},"min":{"1min":0.67500000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.155,"5min":1.353,"15min":1.353},"last":1.0569999999999999}]},{"osd":3,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.95899999999999996},"min":{"1min":0.76200000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1279999999999999,"5min":1.23,"15min":1.23},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.90200000000000002},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.85299999999999998}]},{"osd":4,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82699999999999996,"15min":0.83099999999999996},"min":{"1min":0.53000000000000003,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":1.091},{"interface":"front","average":{"1min":1.006,"5min":0.99099999999999999,"15min":1.0169999999999999},"min":{"1min":0.77900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2649999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Fri Nov 17 06:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.75900000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.002,"5min":1.04,"15min":1.0389999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.246,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.97499999999999998}]},{"osd":6,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0209999999999999,"15min":1.03},"min":{"1min":0.76000000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1719999999999999,"5min":1.377,"15min":1.377},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.099,"15min":1.115},"min":{"1min":0.73999999999999999,"5min":0.627,"15min":0.627},"max":{"1min":1.236,"5min":1.647,"15min":1.647},"last":1.224}]},{"osd":7,"last update":"Fri Nov 17 06:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1659999999999999,"15min":1.1579999999999999},"min":{"1min":0.96299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.294},{"interface":"front","average":{"1min":1.139,"5min":1.161,"15min":1.159},"min":{"1min":0.85499999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4419999999999999,"5min":1.544,"15min":1.544},"last":1.272}]}]},{"osd":0,"up_from":16,"seq":68719476795,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 17 06:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1379999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.83699999999999997},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1000000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.84799999999999998}]},{"osd":2,"last update":"Fri Nov 17 06:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.77100000000000002,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80600000000000005,"15min":0.78200000000000003},"min":{"1min":0.58699999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.135,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.98699999999999999}]},{"osd":3,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95499999999999996,"15min":0.92100000000000004},"min":{"1min":0.74399999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.276,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":1.175},{"interface":"front","average":{"1min":0.999,"5min":0.97499999999999998,"15min":0.94699999999999995},"min":{"1min":0.84099999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.206,"5min":1.222,"15min":1.222},"last":1.0629999999999999}]},{"osd":4,"last update":"Fri Nov 17 06:55:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86799999999999999,"15min":0.88800000000000001},"min":{"1min":0.57899999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.173,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.64000000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.083,"5min":1.355,"15min":1.355},"last":0.92800000000000005}]},{"osd":5,"last update":"Fri Nov 17 06:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.93899999999999995,"15min":0.93300000000000005},"min":{"1min":0.73399999999999999,"5min":0.628,"15min":0.628},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":1.107},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.048,"15min":1.034},"min":{"1min":0.875,"5min":0.62,"15min":0.62},"max":{"1min":1.1419999999999999,"5min":1.587,"15min":1.587},"last":1.1299999999999999}]},{"osd":6,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.022,"15min":1.044},"min":{"1min":0.747,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.45},{"interface":"front","average":{"1min":1.056,"5min":1.0369999999999999,"15min":1.046},"min":{"1min":0.872,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2869999999999999,"5min":1.349,"15min":1.349},"last":1.347}]},{"osd":7,"last update":"Fri Nov 17 06:54:40 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1519999999999999,"15min":1.1579999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1279999999999999,"15min":1.121},"min":{"1min":0.95199999999999996,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.464,"5min":1.7470000000000001,"15min":1.7470000000000001},"last":1.514}]}]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82199999999999995,"15min":0.78800000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1319999999999999,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.80600000000000005,"15min":0.77400000000000002},"min":{"1min":0.66200000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.94699999999999995}]},{"osd":2,"last update":"Fri Nov 17 06:55:15 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.96699999999999997,"15min":0.96299999999999997},"min":{"1min":0.65400000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.042,"5min":0.95899999999999996,"15min":0.91600000000000004},"min":{"1min":0.84299999999999997,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.3180000000000001,"5min":1.357,"15min":1.357},"last":0.97999999999999998}]},{"osd":3,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.78500000000000003,"15min":0.745},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.90900000000000003,"15min":0.86099999999999999},"min":{"1min":0.57699999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.181}]},{"osd":4,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.95199999999999996,"15min":0.93500000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.089},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.87},"min":{"1min":0.65000000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.296,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.0509999999999999}]},{"osd":5,"last update":"Fri Nov 17 06:54:39 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.80400000000000005,"15min":0.78400000000000003},"min":{"1min":0.58899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.879,"15min":0.84399999999999997},"min":{"1min":0.53900000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.417,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.304}]},{"osd":6,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.032,"15min":0.98599999999999999},"min":{"1min":0.73399999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":1.51,"15min":1.51},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.0149999999999999,"15min":0.95199999999999996},"min":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.3879999999999999,"5min":1.538,"15min":1.538},"last":1.2170000000000001}]},{"osd":7,"last update":"Fri Nov 17 06:54:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.042,"15min":1.0449999999999999},"min":{"1min":0.60699999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.4770000000000001,"5min":3.379,"15min":3.379},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0229999999999999,"15min":0.98499999999999999},"min":{"1min":0.66500000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.353}]}]},{"osd":4,"up_from":26,"seq":111669149751,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.55400000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0960000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.751,"15min":0.745},"min":{"1min":0.56299999999999994,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93899999999999995,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.85899999999999999}]},{"osd":1,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.90500000000000003,"15min":0.86199999999999999},"min":{"1min":0.77700000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2,"5min":1.538,"15min":1.538},"last":1.123},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.91100000000000003,"15min":0.90100000000000002},"min":{"1min":0.69999999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":1.048}]},{"osd":2,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81899999999999995,"15min":0.80300000000000005},"min":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.155,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.85499999999999998,"15min":0.84099999999999997},"min":{"1min":0.63200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.123,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.012}]},{"osd":3,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.92900000000000005,"15min":0.90700000000000003},"min":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1890000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96799999999999997,"15min":0.95599999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.1859999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.0720000000000001}]},{"osd":5,"last update":"Fri Nov 17 06:54:38 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.98499999999999999,"15min":0.98899999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.0109999999999999,"15min":0.98599999999999999},"min":{"1min":0.81799999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":1.248}]},{"osd":6,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.995,"15min":0.98399999999999999},"min":{"1min":0.67600000000000005,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3280000000000001,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.032,"15min":1.0269999999999999},"min":{"1min":0.82299999999999995,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.105}]},{"osd":7,"last update":"Fri Nov 17 06:54:42 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.083,"15min":1.0649999999999999},"min":{"1min":0.83499999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.292,"5min":1.377,"15min":1.377},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.135,"15min":1.113},"min":{"1min":0.95299999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.526,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.7130000000000001}]}]},{"osd":3,"up_from":30,"seq":128849018934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1419999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.88200000000000001,"15min":0.86699999999999999},"min":{"1min":0.56799999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.82299999999999995}]},{"osd":1,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68999999999999995,"15min":0.65200000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.129,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.62},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.67200000000000004},"min":{"1min":0.503,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0840000000000001,"5min":1.167,"15min":1.167},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.91500000000000004,"15min":0.92100000000000004},"min":{"1min":0.629,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.054,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.91200000000000003,"15min":0.90800000000000003},"min":{"1min":0.749,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.90100000000000002}]},{"osd":4,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.90300000000000002,"15min":0.875},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1919999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.96899999999999997,"15min":0.95899999999999996},"min":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2090000000000001,"5min":1.254,"15min":1.254},"last":0.84499999999999997}]},{"osd":5,"last update":"Fri Nov 17 06:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80100000000000005,"15min":0.76100000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1359999999999999,"5min":1.585,"15min":1.585},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.87,"15min":0.81399999999999995},"min":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.95699999999999996}]},{"osd":6,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.071},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0189999999999999,"15min":0.98999999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.3340000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":1.1799999999999999}]},{"osd":7,"last update":"Fri Nov 17 06:54:41 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.95099999999999996,"15min":0.93700000000000006},"min":{"1min":0.48799999999999999,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.98999999999999999,"15min":0.96599999999999997},"min":{"1min":0.64800000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.1319999999999999}]}]},{"osd":6,"up_from":30,"seq":128849018935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.036,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.90100000000000002,"15min":0.92100000000000004},"min":{"1min":0.71499999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.1859999999999999,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":0.88800000000000001}]},{"osd":1,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.97999999999999998,"15min":0.97499999999999998},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.129,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.93799999999999994},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.21,"5min":2.024,"15min":2.024},"last":0.93200000000000005}]},{"osd":2,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82899999999999996,"15min":0.83899999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0249999999999999,"5min":1.925,"15min":1.925},"last":0.63},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84599999999999997,"15min":0.878},"min":{"1min":0.57999999999999996,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1259999999999999,"5min":3.141,"15min":3.141},"last":0.66200000000000003}]},{"osd":3,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95199999999999996,"15min":0.96599999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1279999999999999,"5min":2.9550000000000001,"15min":2.9550000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.73199999999999998,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.1779999999999999,"5min":2.823,"15min":2.823},"last":1.0509999999999999}]},{"osd":4,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.96299999999999997,"15min":0.98999999999999999},"min":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1399999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.04,"15min":1.0840000000000001},"min":{"1min":0.67900000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.282,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":0.95599999999999996}]},{"osd":5,"last update":"Fri Nov 17 06:54:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.125,"15min":1.147},"min":{"1min":0.89500000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.476,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.1539999999999999,"15min":1.177},"min":{"1min":0.88800000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.274,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":1.0820000000000001}]},{"osd":7,"last update":"Fri Nov 17 06:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.151},"min":{"1min":0.87,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.2490000000000001,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.196,"15min":1.22},"min":{"1min":0.89900000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.49,"5min":3.1829999999999998,"15min":3.1829999999999998},"last":1.1200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T06:55:42.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph pg dump --format=json 2023-11-17T06:55:42.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:42 smithi193 bash[28559]: cluster 2023-11-17T06:55:41.396950+0000 mgr.smithi097.swvonl (mgr.14180) 257 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:42.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:42 smithi193 bash[28559]: audit 2023-11-17T06:55:41.576938+0000 mgr.smithi097.swvonl (mgr.14180) 258 : audit [DBG] from='client.14462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:55:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:44 smithi097 bash[24958]: cluster 2023-11-17T06:55:43.397503+0000 mgr.smithi097.swvonl (mgr.14180) 259 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:44.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:44 smithi193 bash[28559]: cluster 2023-11-17T06:55:43.397503+0000 mgr.smithi097.swvonl (mgr.14180) 259 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:46.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:46 smithi097 bash[24958]: cluster 2023-11-17T06:55:45.398106+0000 mgr.smithi097.swvonl (mgr.14180) 260 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:46.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:46 smithi193 bash[28559]: cluster 2023-11-17T06:55:45.398106+0000 mgr.smithi097.swvonl (mgr.14180) 260 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:55:47.219 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:55:48.706 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:48 smithi097 bash[24958]: cluster 2023-11-17T06:55:47.398671+0000 mgr.smithi097.swvonl (mgr.14180) 261 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.071 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:48 smithi097 bash[24958]: audit 2023-11-17T06:55:47.991139+0000 mon.smithi097 (mon.0) 607 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:56:42.072 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:50 smithi097 bash[24958]: cluster 2023-11-17T06:55:49.399330+0000 mgr.smithi097.swvonl (mgr.14180) 262 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.074 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:56:42.088 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2023-11-17T06:56:42.088 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2023-11-17T06:55:49.399025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749928448,"kb_used":2379964,"kb_used_data":3196,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747548484,"statfs":{"total":767926730752,"available":765489647616,"internally_reserved":0,"allocated":3272704,"data_stored":1276661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-11-17T06:51:39.449751+0000","last_change":"2023-11-17T06:51:39.449751+0000","last_active":"2023-11-17T06:51:39.449751+0000","last_peered":"2023-11-17T06:51:39.449751+0000","last_clean":"2023-11-17T06:51:39.449751+0000","last_became_active":"2023-11-17T06:51:39.449303+0000","last_became_peered":"2023-11-17T06:51:39.449303+0000","last_unstale":"2023-11-17T06:51:39.449751+0000","last_undegraded":"2023-11-17T06:51:39.449751+0000","last_fullsized":"2023-11-17T06:51:39.449751+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-17T06:51:11.613640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-17T06:51:11.613640+0000","last_clean_scrub_stamp":"2023-11-17T06:51:11.613640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":39,"seq":167503724597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297492,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443564,"statfs":{"total":95990841344,"available":95686209536,"internally_reserved":0,"allocated":405504,"data_stored":159307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.84799999999999998,"15min":0.85699999999999998},"min":{"1min":0.64400000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.98999999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89200000000000002,"15min":0.88600000000000001},"min":{"1min":0.627,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0940000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.70699999999999996}]},{"osd":1,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.69599999999999995,"15min":0.67400000000000004},"min":{"1min":0.56499999999999995,"5min":0.312,"15min":0.312},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.73699999999999999,"15min":0.69599999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.876}]},{"osd":2,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78800000000000003,"15min":0.78000000000000003},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.0269999999999999,"5min":1.26,"15min":1.26},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.88,"5min":0.89900000000000002,"15min":0.89500000000000002},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1160000000000001,"5min":1.226,"15min":1.226},"last":0.91300000000000003}]},{"osd":3,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.82699999999999996,"15min":0.80300000000000005},"min":{"1min":0.47999999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79000000000000004,"15min":0.77000000000000002},"min":{"1min":0.49099999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.68999999999999995}]},{"osd":4,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.92000000000000004,"15min":0.91500000000000004},"min":{"1min":0.70299999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.135,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0049999999999999,"15min":1.004},"min":{"1min":0.68200000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1539999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.99399999999999999}]},{"osd":5,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.99399999999999999},"min":{"1min":0.503,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.198,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":1.044},{"interface":"front","average":{"1min":1.05,"5min":1.0149999999999999,"15min":1.006},"min":{"1min":0.51400000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.276}]},{"osd":6,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.1140000000000001,"15min":1.0980000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0489999999999999,"15min":1.0489999999999999},"min":{"1min":0.83199999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5509999999999999,"5min":1.766,"15min":1.766},"last":1.123}]}]},{"osd":5,"up_from":35,"seq":150323855415,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.67100000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.748},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.92800000000000005,"15min":0.91800000000000004},"min":{"1min":0.71499999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.78900000000000003}]},{"osd":1,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.70999999999999996,"15min":0.68200000000000005},"min":{"1min":0.47899999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.03,"5min":1.133,"15min":1.133},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70599999999999996,"15min":0.68000000000000005},"min":{"1min":0.495,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.98099999999999998,"5min":1.212,"15min":1.212},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.90600000000000003,"15min":0.91600000000000004},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.61499999999999999,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.248,"5min":1.248,"15min":1.248},"last":0.67100000000000004}]},{"osd":3,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.51800000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1599999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.80700000000000005},"min":{"1min":0.57699999999999996,"5min":0.436,"15min":0.436},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.98299999999999998,"15min":0.98999999999999999},"min":{"1min":0.748,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.169,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.81499999999999995,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.2909999999999999,"5min":1.385,"15min":1.385},"last":0.83999999999999997}]},{"osd":6,"last update":"Fri Nov 17 06:55:34 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":1.0049999999999999,"15min":1.01},"min":{"1min":0.67400000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.1950000000000001,"5min":1.276,"15min":1.276},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.026,"15min":1.0349999999999999},"min":{"1min":0.80200000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.1459999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.80600000000000005}]},{"osd":7,"last update":"Fri Nov 17 06:54:47 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.98099999999999998,"15min":0.97299999999999998},"min":{"1min":0.68200000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.98999999999999999,"5min":1.0209999999999999,"15min":1.0149999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2609999999999999,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.754}]}]},{"osd":2,"up_from":20,"seq":85899345979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:12 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72299999999999998,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.216,"5min":1.276,"15min":1.276},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.69399999999999995},"min":{"1min":0.59599999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.85799999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.72899999999999998}]},{"osd":1,"last update":"Fri Nov 17 06:55:16 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85299999999999998,"15min":0.83599999999999997},"min":{"1min":0.70499999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1160000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93100000000000005,"15min":0.91500000000000004},"min":{"1min":0.67500000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.155,"5min":1.353,"15min":1.353},"last":1.006}]},{"osd":3,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.95899999999999996},"min":{"1min":0.76200000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1279999999999999,"5min":1.23,"15min":1.23},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.92000000000000004,"15min":0.90200000000000002},"min":{"1min":0.626,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":1.026}]},{"osd":4,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82699999999999996,"15min":0.83099999999999996},"min":{"1min":0.53000000000000003,"5min":0.439,"15min":0.439},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.006,"5min":0.99099999999999999,"15min":1.0169999999999999},"min":{"1min":0.77900000000000003,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2649999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.878}]},{"osd":5,"last update":"Fri Nov 17 06:55:36 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.75900000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.002,"5min":1.04,"15min":1.0389999999999999},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.246,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.105}]},{"osd":6,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0209999999999999,"15min":1.03},"min":{"1min":0.76000000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1719999999999999,"5min":1.377,"15min":1.377},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.099,"15min":1.115},"min":{"1min":0.73999999999999999,"5min":0.627,"15min":0.627},"max":{"1min":1.236,"5min":1.647,"15min":1.647},"last":1.1359999999999999}]},{"osd":7,"last update":"Fri Nov 17 06:54:45 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1659999999999999,"15min":1.1579999999999999},"min":{"1min":0.96299999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.139,"5min":1.161,"15min":1.159},"min":{"1min":0.85499999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.4419999999999999,"5min":1.544,"15min":1.544},"last":1.3300000000000001}]}]},{"osd":0,"up_from":16,"seq":68719476797,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 17 06:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.86499999999999999,"15min":0.85899999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1379999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.85399999999999998,"15min":0.83699999999999997},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1000000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Fri Nov 17 06:55:18 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.77100000000000002,"15min":0.752},"min":{"1min":0.58199999999999996,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.80600000000000005,"15min":0.78200000000000003},"min":{"1min":0.58699999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.135,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.753}]},{"osd":3,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95499999999999996,"15min":0.92100000000000004},"min":{"1min":0.74399999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.276,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":1.04},{"interface":"front","average":{"1min":0.999,"5min":0.97499999999999998,"15min":0.94699999999999995},"min":{"1min":0.84099999999999997,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.206,"5min":1.222,"15min":1.222},"last":0.92200000000000004}]},{"osd":4,"last update":"Fri Nov 17 06:55:22 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86799999999999999,"15min":0.88800000000000001},"min":{"1min":0.57899999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.173,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.88},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.91900000000000004,"15min":0.93500000000000005},"min":{"1min":0.64000000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.083,"5min":1.355,"15min":1.355},"last":0.85199999999999998}]},{"osd":5,"last update":"Fri Nov 17 06:55:35 2023","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.93899999999999995,"15min":0.93300000000000005},"min":{"1min":0.73399999999999999,"5min":0.628,"15min":0.628},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":1.01},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.048,"15min":1.034},"min":{"1min":0.875,"5min":0.62,"15min":0.62},"max":{"1min":1.1419999999999999,"5min":1.587,"15min":1.587},"last":0.996}]},{"osd":6,"last update":"Fri Nov 17 06:55:28 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.022,"15min":1.044},"min":{"1min":0.747,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.0369999999999999,"15min":1.046},"min":{"1min":0.872,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2869999999999999,"5min":1.349,"15min":1.349},"last":1.3220000000000001}]},{"osd":7,"last update":"Fri Nov 17 06:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1459999999999999,"15min":1.1559999999999999},"min":{"1min":0.88500000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.4810000000000001,"5min":1.585,"15min":1.585},"last":1.109},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.141,"15min":1.125},"min":{"1min":0.98299999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"last":1.2290000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:15 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82199999999999995,"15min":0.78800000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.1319999999999999,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.80600000000000005,"15min":0.77400000000000002},"min":{"1min":0.66200000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Fri Nov 17 06:55:15 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":0.96699999999999997,"15min":0.96299999999999997},"min":{"1min":0.65400000000000003,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.169},{"interface":"front","average":{"1min":1.042,"5min":0.95899999999999996,"15min":0.91600000000000004},"min":{"1min":0.84299999999999997,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.3180000000000001,"5min":1.357,"15min":1.357},"last":1.0029999999999999}]},{"osd":3,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.78500000000000003,"15min":0.745},"min":{"1min":0.60999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.90900000000000003,"15min":0.86099999999999999},"min":{"1min":0.57699999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.216}]},{"osd":4,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.95199999999999996,"15min":0.93500000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.752},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.87},"min":{"1min":0.65000000000000002,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.296,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Fri Nov 17 06:55:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81299999999999994,"15min":0.78700000000000003},"min":{"1min":0.48399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1040000000000001,"5min":1.151,"15min":1.151},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.90900000000000003,"15min":0.85399999999999998},"min":{"1min":0.53600000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.304,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":1.0269999999999999}]},{"osd":6,"last update":"Fri Nov 17 06:55:23 2023","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.032,"15min":0.98599999999999999},"min":{"1min":0.73399999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.3440000000000001,"5min":1.51,"15min":1.51},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.0149999999999999,"15min":0.95199999999999996},"min":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.3879999999999999,"5min":1.538,"15min":1.538},"last":0.95899999999999996}]},{"osd":7,"last update":"Fri Nov 17 06:55:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0349999999999999,"15min":1.0429999999999999},"min":{"1min":0.67600000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1970000000000001,"5min":3.379,"15min":3.379},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.05,"15min":0.99399999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.353,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.92500000000000004}]}]},{"osd":4,"up_from":26,"seq":111669149752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.70299999999999996,"15min":0.68799999999999994},"min":{"1min":0.55400000000000005,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0960000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.751,"15min":0.745},"min":{"1min":0.56299999999999994,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93899999999999995,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.84699999999999998}]},{"osd":1,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.90500000000000003,"15min":0.86199999999999999},"min":{"1min":0.77700000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2,"5min":1.538,"15min":1.538},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.91100000000000003,"15min":0.90100000000000002},"min":{"1min":0.69999999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.0900000000000001,"5min":1.365,"15min":1.365},"last":0.95499999999999996}]},{"osd":2,"last update":"Fri Nov 17 06:55:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81899999999999995,"15min":0.80300000000000005},"min":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.155,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.038},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.85499999999999998,"15min":0.84099999999999997},"min":{"1min":0.63200000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.123,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":1.0940000000000001}]},{"osd":3,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.92900000000000005,"15min":0.90700000000000003},"min":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.1890000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96799999999999997,"15min":0.95599999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.1859999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":1.2170000000000001}]},{"osd":5,"last update":"Fri Nov 17 06:55:40 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.99199999999999999,"15min":0.99099999999999999},"min":{"1min":0.82999999999999996,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.2050000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0309999999999999,"15min":0.99299999999999999},"min":{"1min":0.78700000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.2549999999999999,"5min":1.365,"15min":1.365},"last":1.1579999999999999}]},{"osd":6,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.995,"15min":0.98399999999999999},"min":{"1min":0.67600000000000005,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3280000000000001,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.032,"15min":1.0269999999999999},"min":{"1min":0.82299999999999995,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Nov 17 06:54:42 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.083,"15min":1.0649999999999999},"min":{"1min":0.83499999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.292,"5min":1.377,"15min":1.377},"last":1.125},{"interface":"front","average":{"1min":1.155,"5min":1.135,"15min":1.113},"min":{"1min":0.95299999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.526,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.202}]}]},{"osd":3,"up_from":30,"seq":128849018935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1419999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.88200000000000001,"15min":0.86699999999999999},"min":{"1min":0.56799999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.92500000000000004}]},{"osd":1,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.68999999999999995,"15min":0.65200000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.129,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.69799999999999995,"15min":0.67200000000000004},"min":{"1min":0.503,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0840000000000001,"5min":1.167,"15min":1.167},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.91500000000000004,"15min":0.92100000000000004},"min":{"1min":0.629,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.054,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.91200000000000003,"15min":0.90800000000000003},"min":{"1min":0.749,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":1.0389999999999999}]},{"osd":4,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.90300000000000002,"15min":0.875},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1919999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.96899999999999997,"15min":0.95899999999999996},"min":{"1min":0.66500000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2090000000000001,"5min":1.254,"15min":1.254},"last":0.89800000000000002}]},{"osd":5,"last update":"Fri Nov 17 06:55:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.80100000000000005,"15min":0.76100000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1359999999999999,"5min":1.585,"15min":1.585},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.87,"15min":0.81399999999999995},"min":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.21,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Fri Nov 17 06:55:25 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0189999999999999,"15min":0.98999999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.3340000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":1.0189999999999999}]},{"osd":7,"last update":"Fri Nov 17 06:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":0.96899999999999997,"15min":0.94299999999999995},"min":{"1min":0.59299999999999997,"5min":0.439,"15min":0.439},"max":{"1min":1.268,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.014,"15min":0.97399999999999998},"min":{"1min":0.55100000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3440000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.083}]}]},{"osd":6,"up_from":30,"seq":128849018936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93741056,"kb_used":297496,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93443560,"statfs":{"total":95990841344,"available":95686205440,"internally_reserved":0,"allocated":409600,"data_stored":159622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79100000000000004,"15min":0.81299999999999994},"min":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.036,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.874,"5min":0.90100000000000002,"15min":0.92100000000000004},"min":{"1min":0.71499999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.1859999999999999,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":0.92900000000000005}]},{"osd":1,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.97999999999999998,"15min":0.97499999999999998},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.129,"5min":2.8420000000000001,"15min":2.8420000000000001},"last":1.002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94799999999999995,"15min":0.93799999999999994},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.21,"5min":2.024,"15min":2.024},"last":0.97899999999999998}]},{"osd":2,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.82899999999999996,"15min":0.83899999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0249999999999999,"5min":1.925,"15min":1.925},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84599999999999997,"15min":0.878},"min":{"1min":0.57999999999999996,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1259999999999999,"5min":3.141,"15min":3.141},"last":1.022}]},{"osd":3,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.95199999999999996,"15min":0.96599999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1279999999999999,"5min":2.9550000000000001,"15min":2.9550000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.73199999999999998,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.1779999999999999,"5min":2.823,"15min":2.823},"last":0.78800000000000003}]},{"osd":4,"last update":"Fri Nov 17 06:55:24 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.96299999999999997,"15min":0.98999999999999999},"min":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1399999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.04,"15min":1.0840000000000001},"min":{"1min":0.67900000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.282,"5min":2.8959999999999999,"15min":2.8959999999999999},"last":0.89700000000000002}]},{"osd":5,"last update":"Fri Nov 17 06:55:42 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.1000000000000001,"15min":1.139},"min":{"1min":0.76700000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.631,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.131,"15min":1.169},"min":{"1min":0.76800000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.601,"5min":2.9260000000000002,"15min":2.9260000000000002},"last":1.044}]},{"osd":7,"last update":"Fri Nov 17 06:54:44 2023","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.1259999999999999,"15min":1.151},"min":{"1min":0.87,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.2490000000000001,"5min":2.7410000000000001,"15min":2.7410000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.196,"15min":1.22},"min":{"1min":0.89900000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.49,"5min":3.1829999999999998,"15min":3.1829999999999998},"last":1.077}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-17T06:56:42.138 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-17T06:56:42.138 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-17T06:56:42.141 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-17T06:56:42.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph health --format=json 2023-11-17T06:56:42.144 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:48 smithi193 bash[28559]: cluster 2023-11-17T06:55:47.398671+0000 mgr.smithi097.swvonl (mgr.14180) 261 : cluster [DBG] pgmap v243: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.144 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:48 smithi193 bash[28559]: audit 2023-11-17T06:55:47.991139+0000 mon.smithi097 (mon.0) 607 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:56:42.144 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:50 smithi193 bash[28559]: cluster 2023-11-17T06:55:49.399330+0000 mgr.smithi097.swvonl (mgr.14180) 262 : cluster [DBG] pgmap v244: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:51 smithi193 bash[28559]: audit 2023-11-17T06:55:50.697781+0000 mgr.smithi097.swvonl (mgr.14180) 263 : audit [DBG] from='client.14466 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:52 smithi193 bash[28559]: cluster 2023-11-17T06:55:51.399942+0000 mgr.smithi097.swvonl (mgr.14180) 264 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: cluster 2023-11-17T06:55:53.400447+0000 mgr.smithi097.swvonl (mgr.14180) 265 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:53.513377+0000 mon.smithi097 (mon.0) 608 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:54.129108+0000 mon.smithi097 (mon.0) 609 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:54.434319+0000 mon.smithi097 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:54.435689+0000 mon.smithi097 (mon.0) 611 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:54.443693+0000 mon.smithi097 (mon.0) 612 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:54 smithi193 bash[28559]: audit 2023-11-17T06:55:54.451874+0000 mon.smithi097 (mon.0) 613 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:56 smithi193 bash[28559]: cluster 2023-11-17T06:55:55.400985+0000 mgr.smithi097.swvonl (mgr.14180) 266 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:55:58 smithi193 bash[28559]: cluster 2023-11-17T06:55:57.401514+0000 mgr.smithi097.swvonl (mgr.14180) 267 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:00 smithi193 bash[28559]: cluster 2023-11-17T06:55:59.402095+0000 mgr.smithi097.swvonl (mgr.14180) 268 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.145 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:02 smithi193 bash[28559]: cluster 2023-11-17T06:56:01.402685+0000 mgr.smithi097.swvonl (mgr.14180) 269 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:04 smithi193 bash[28559]: cluster 2023-11-17T06:56:03.403266+0000 mgr.smithi097.swvonl (mgr.14180) 270 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:06 smithi193 bash[28559]: cluster 2023-11-17T06:56:05.403970+0000 mgr.smithi097.swvonl (mgr.14180) 271 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:08 smithi193 bash[28559]: cluster 2023-11-17T06:56:07.404580+0000 mgr.smithi097.swvonl (mgr.14180) 272 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:10 smithi193 bash[28559]: cluster 2023-11-17T06:56:09.405167+0000 mgr.smithi097.swvonl (mgr.14180) 273 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:12 smithi193 bash[28559]: cluster 2023-11-17T06:56:11.405670+0000 mgr.smithi097.swvonl (mgr.14180) 274 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:14 smithi193 bash[28559]: cluster 2023-11-17T06:56:13.406312+0000 mgr.smithi097.swvonl (mgr.14180) 275 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:16 smithi193 bash[28559]: cluster 2023-11-17T06:56:15.406848+0000 mgr.smithi097.swvonl (mgr.14180) 276 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:18 smithi193 bash[28559]: cluster 2023-11-17T06:56:17.407353+0000 mgr.smithi097.swvonl (mgr.14180) 277 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:20 smithi193 bash[28559]: cluster 2023-11-17T06:56:19.407900+0000 mgr.smithi097.swvonl (mgr.14180) 278 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:22 smithi193 bash[28559]: cluster 2023-11-17T06:56:21.408577+0000 mgr.smithi097.swvonl (mgr.14180) 279 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:24 smithi193 bash[28559]: cluster 2023-11-17T06:56:23.409167+0000 mgr.smithi097.swvonl (mgr.14180) 280 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:26 smithi193 bash[28559]: cluster 2023-11-17T06:56:25.409843+0000 mgr.smithi097.swvonl (mgr.14180) 281 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:28 smithi193 bash[28559]: cluster 2023-11-17T06:56:27.410398+0000 mgr.smithi097.swvonl (mgr.14180) 282 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:30 smithi193 bash[28559]: cluster 2023-11-17T06:56:29.411017+0000 mgr.smithi097.swvonl (mgr.14180) 283 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:32 smithi193 bash[28559]: cluster 2023-11-17T06:56:31.411580+0000 mgr.smithi097.swvonl (mgr.14180) 284 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:34 smithi193 bash[28559]: cluster 2023-11-17T06:56:33.412161+0000 mgr.smithi097.swvonl (mgr.14180) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:36 smithi193 bash[28559]: cluster 2023-11-17T06:56:35.412736+0000 mgr.smithi097.swvonl (mgr.14180) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:38 smithi193 bash[28559]: cluster 2023-11-17T06:56:37.413273+0000 mgr.smithi097.swvonl (mgr.14180) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:40 smithi193 bash[28559]: cluster 2023-11-17T06:56:39.413861+0000 mgr.smithi097.swvonl (mgr.14180) 288 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:51 smithi097 bash[24958]: audit 2023-11-17T06:55:50.697781+0000 mgr.smithi097.swvonl (mgr.14180) 263 : audit [DBG] from='client.14466 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:56:42.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:52 smithi097 bash[24958]: cluster 2023-11-17T06:55:51.399942+0000 mgr.smithi097.swvonl (mgr.14180) 264 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: cluster 2023-11-17T06:55:53.400447+0000 mgr.smithi097.swvonl (mgr.14180) 265 : cluster [DBG] pgmap v246: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:53.513377+0000 mon.smithi097 (mon.0) 608 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:54.129108+0000 mon.smithi097 (mon.0) 609 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:54.434319+0000 mon.smithi097 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:54.435689+0000 mon.smithi097 (mon.0) 611 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:54.443693+0000 mon.smithi097 (mon.0) 612 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:54 smithi097 bash[24958]: audit 2023-11-17T06:55:54.451874+0000 mon.smithi097 (mon.0) 613 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:56 smithi097 bash[24958]: cluster 2023-11-17T06:55:55.400985+0000 mgr.smithi097.swvonl (mgr.14180) 266 : cluster [DBG] pgmap v247: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:55:58 smithi097 bash[24958]: cluster 2023-11-17T06:55:57.401514+0000 mgr.smithi097.swvonl (mgr.14180) 267 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:00 smithi097 bash[24958]: cluster 2023-11-17T06:55:59.402095+0000 mgr.smithi097.swvonl (mgr.14180) 268 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:02 smithi097 bash[24958]: cluster 2023-11-17T06:56:01.402685+0000 mgr.smithi097.swvonl (mgr.14180) 269 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:04 smithi097 bash[24958]: cluster 2023-11-17T06:56:03.403266+0000 mgr.smithi097.swvonl (mgr.14180) 270 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:06 smithi097 bash[24958]: cluster 2023-11-17T06:56:05.403970+0000 mgr.smithi097.swvonl (mgr.14180) 271 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:08 smithi097 bash[24958]: cluster 2023-11-17T06:56:07.404580+0000 mgr.smithi097.swvonl (mgr.14180) 272 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:10 smithi097 bash[24958]: cluster 2023-11-17T06:56:09.405167+0000 mgr.smithi097.swvonl (mgr.14180) 273 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:12 smithi097 bash[24958]: cluster 2023-11-17T06:56:11.405670+0000 mgr.smithi097.swvonl (mgr.14180) 274 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:14 smithi097 bash[24958]: cluster 2023-11-17T06:56:13.406312+0000 mgr.smithi097.swvonl (mgr.14180) 275 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:16 smithi097 bash[24958]: cluster 2023-11-17T06:56:15.406848+0000 mgr.smithi097.swvonl (mgr.14180) 276 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:18 smithi097 bash[24958]: cluster 2023-11-17T06:56:17.407353+0000 mgr.smithi097.swvonl (mgr.14180) 277 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:20 smithi097 bash[24958]: cluster 2023-11-17T06:56:19.407900+0000 mgr.smithi097.swvonl (mgr.14180) 278 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:22 smithi097 bash[24958]: cluster 2023-11-17T06:56:21.408577+0000 mgr.smithi097.swvonl (mgr.14180) 279 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:24 smithi097 bash[24958]: cluster 2023-11-17T06:56:23.409167+0000 mgr.smithi097.swvonl (mgr.14180) 280 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.150 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:26 smithi097 bash[24958]: cluster 2023-11-17T06:56:25.409843+0000 mgr.smithi097.swvonl (mgr.14180) 281 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:28 smithi097 bash[24958]: cluster 2023-11-17T06:56:27.410398+0000 mgr.smithi097.swvonl (mgr.14180) 282 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:30 smithi097 bash[24958]: cluster 2023-11-17T06:56:29.411017+0000 mgr.smithi097.swvonl (mgr.14180) 283 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:32 smithi097 bash[24958]: cluster 2023-11-17T06:56:31.411580+0000 mgr.smithi097.swvonl (mgr.14180) 284 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:34 smithi097 bash[24958]: cluster 2023-11-17T06:56:33.412161+0000 mgr.smithi097.swvonl (mgr.14180) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:36 smithi097 bash[24958]: cluster 2023-11-17T06:56:35.412736+0000 mgr.smithi097.swvonl (mgr.14180) 286 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:38 smithi097 bash[24958]: cluster 2023-11-17T06:56:37.413273+0000 mgr.smithi097.swvonl (mgr.14180) 287 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.171 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:40 smithi097 bash[24958]: cluster 2023-11-17T06:56:39.413861+0000 mgr.smithi097.swvonl (mgr.14180) 288 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:42 smithi097 bash[24958]: cluster 2023-11-17T06:56:41.414449+0000 mgr.smithi097.swvonl (mgr.14180) 289 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:42.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:42 smithi193 bash[28559]: cluster 2023-11-17T06:56:41.414449+0000 mgr.smithi097.swvonl (mgr.14180) 289 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:44 smithi097 bash[24958]: cluster 2023-11-17T06:56:43.414980+0000 mgr.smithi097.swvonl (mgr.14180) 290 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:44.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:44 smithi193 bash[28559]: cluster 2023-11-17T06:56:43.414980+0000 mgr.smithi097.swvonl (mgr.14180) 290 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:46.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:46 smithi097 bash[24958]: cluster 2023-11-17T06:56:45.415534+0000 mgr.smithi097.swvonl (mgr.14180) 291 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:46.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:46 smithi193 bash[28559]: cluster 2023-11-17T06:56:45.415534+0000 mgr.smithi097.swvonl (mgr.14180) 291 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:46.956 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/mon.smithi097/config 2023-11-17T06:56:48.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:48 smithi097 bash[24958]: cluster 2023-11-17T06:56:47.416165+0000 mgr.smithi097.swvonl (mgr.14180) 292 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:48.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:48 smithi193 bash[28559]: cluster 2023-11-17T06:56:47.416165+0000 mgr.smithi097.swvonl (mgr.14180) 292 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:50.509 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:56:50.749 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:50 smithi097 bash[24958]: cluster 2023-11-17T06:56:49.416787+0000 mgr.smithi097.swvonl (mgr.14180) 293 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:50.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:50 smithi193 bash[28559]: cluster 2023-11-17T06:56:49.416787+0000 mgr.smithi097.swvonl (mgr.14180) 293 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:51.405 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-17T06:56:51.406 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-17T06:56:51.406 INFO:tasks.cephadm:Setup complete, yielding 2023-11-17T06:56:51.406 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T06:56:51.415 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T06:56:51.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-17T06:56:51.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:51 smithi097 bash[24958]: audit 2023-11-17T06:56:50.506362+0000 mon.smithi097 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.97:0/1456782213' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T06:56:51.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:51 smithi193 bash[28559]: audit 2023-11-17T06:56:50.506362+0000 mon.smithi097 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.97:0/1456782213' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-17T06:56:52.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:52 smithi097 bash[24958]: cluster 2023-11-17T06:56:51.417364+0000 mgr.smithi097.swvonl (mgr.14180) 294 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:52.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:52 smithi193 bash[28559]: cluster 2023-11-17T06:56:51.417364+0000 mgr.smithi097.swvonl (mgr.14180) 294 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:54.759 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:54 smithi097 bash[24958]: cluster 2023-11-17T06:56:53.417968+0000 mgr.smithi097.swvonl (mgr.14180) 295 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:54.759 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:54 smithi097 bash[24958]: audit 2023-11-17T06:56:54.463430+0000 mon.smithi097 (mon.0) 615 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:56:54.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:54 smithi193 bash[28559]: cluster 2023-11-17T06:56:53.417968+0000 mgr.smithi097.swvonl (mgr.14180) 295 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:54.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:54 smithi193 bash[28559]: audit 2023-11-17T06:56:54.463430+0000 mon.smithi097 (mon.0) 615 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:56:55.129 INFO:teuthology.orchestra.run.smithi097.stdout:Backend: cephadm 2023-11-17T06:56:55.129 INFO:teuthology.orchestra.run.smithi097.stdout:Available: Yes 2023-11-17T06:56:55.130 INFO:teuthology.orchestra.run.smithi097.stdout:Paused: No 2023-11-17T06:56:55.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-17T06:56:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:56 smithi097 bash[24958]: audit 2023-11-17T06:56:55.125829+0000 mgr.smithi097.swvonl (mgr.14180) 296 : audit [DBG] from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:56:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:56 smithi097 bash[24958]: cluster 2023-11-17T06:56:55.418609+0000 mgr.smithi097.swvonl (mgr.14180) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:56.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:56 smithi193 bash[28559]: audit 2023-11-17T06:56:55.125829+0000 mgr.smithi097.swvonl (mgr.14180) 296 : audit [DBG] from='client.14474 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:56:56.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:56 smithi193 bash[28559]: cluster 2023-11-17T06:56:55.418609+0000 mgr.smithi097.swvonl (mgr.14180) 297 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:58.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:56:58 smithi097 bash[24958]: cluster 2023-11-17T06:56:57.419126+0000 mgr.smithi097.swvonl (mgr.14180) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:56:58 smithi193 bash[28559]: cluster 2023-11-17T06:56:57.419126+0000 mgr.smithi097.swvonl (mgr.14180) 298 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.smithi097 smithi097 *:9093,9094 running (7m) 65s ago 8m 12.7M - 0.23.0 ba2b418f427c 9a707423d62e 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:crash.smithi097 smithi097 running (8m) 65s ago 8m 6943k - 16.2.14-171-ged3c8e92 6be2081e30e3 b467c7366a95 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:crash.smithi193 smithi193 running (7m) 65s ago 7m 7079k - 16.2.14-171-ged3c8e92 6be2081e30e3 b303332086f1 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.smithi097 smithi097 *:3000 running (7m) 65s ago 8m 41.5M - 8.3.5 dad864ee21e9 618681e43189 2023-11-17T06:56:59.379 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.smithi097.swvonl smithi097 *:9283 running (10m) 65s ago 10m 430M - 16.2.14-171-ged3c8e92 6be2081e30e3 4fa56981b469 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.smithi193.zqrhit smithi193 *:8443,9283 running (7m) 65s ago 7m 375M - 16.2.14-171-ged3c8e92 6be2081e30e3 d6651d013ae5 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:mon.smithi097 smithi097 running (10m) 65s ago 10m 43.2M 2048M 16.2.14-171-ged3c8e92 6be2081e30e3 480873b37589 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:mon.smithi193 smithi193 running (7m) 65s ago 7m 34.1M 2048M 16.2.14-171-ged3c8e92 6be2081e30e3 689649c876e1 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.smithi097 smithi097 *:9100 running (8m) 65s ago 8m 10.2M - 1.3.1 1dbe0e931976 fe7bc540666f 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.smithi193 smithi193 *:9100 running (7m) 65s ago 7m 9803k - 1.3.1 1dbe0e931976 13ee7933f034 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi193 running (6m) 65s ago 6m 33.5M 4028M 16.2.14-171-ged3c8e92 6be2081e30e3 1fbb2dde7780 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (5m) 65s ago 6m 33.8M 3260M 16.2.14-171-ged3c8e92 6be2081e30e3 3b48a4a69ce0 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi193 running (5m) 65s ago 6m 35.2M 4028M 16.2.14-171-ged3c8e92 6be2081e30e3 6082b9228645 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (5m) 65s ago 5m 33.3M 3260M 16.2.14-171-ged3c8e92 6be2081e30e3 7ff6275c42cd 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi193 running (5m) 65s ago 5m 33.4M 4028M 16.2.14-171-ged3c8e92 6be2081e30e3 253108f364d6 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi097 running (5m) 65s ago 5m 32.4M 3260M 16.2.14-171-ged3c8e92 6be2081e30e3 bc57e743429b 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi193 running (5m) 65s ago 5m 33.5M 4028M 16.2.14-171-ged3c8e92 6be2081e30e3 4d8056a5667c 2023-11-17T06:56:59.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi097 running (5m) 65s ago 5m 33.6M 3260M 16.2.14-171-ged3c8e92 6be2081e30e3 0972ca8513e2 2023-11-17T06:56:59.381 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.smithi097 smithi097 *:9095 running (6m) 65s ago 8m 55.7M - 2.33.4 514e6a882f6e eeee245c2480 2023-11-17T06:57:00.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-17T06:57:00.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:00 smithi097 bash[24958]: audit 2023-11-17T06:56:59.357946+0000 mgr.smithi097.swvonl (mgr.14180) 299 : audit [DBG] from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:00.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:00 smithi097 bash[24958]: cluster 2023-11-17T06:56:59.419783+0000 mgr.smithi097.swvonl (mgr.14180) 300 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:00.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:00 smithi097 bash[24958]: audit 2023-11-17T06:57:00.003309+0000 mon.smithi097 (mon.0) 616 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:00.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:00 smithi193 bash[28559]: audit 2023-11-17T06:56:59.357946+0000 mgr.smithi097.swvonl (mgr.14180) 299 : audit [DBG] from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:00.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:00 smithi193 bash[28559]: cluster 2023-11-17T06:56:59.419783+0000 mgr.smithi097.swvonl (mgr.14180) 300 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:00 smithi193 bash[28559]: audit 2023-11-17T06:57:00.003309+0000 mon.smithi097 (mon.0) 616 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:02.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:01 smithi097 bash[24958]: audit 2023-11-17T06:57:00.658426+0000 mon.smithi097 (mon.0) 617 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:02.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:01 smithi097 bash[24958]: audit 2023-11-17T06:57:00.924827+0000 mon.smithi097 (mon.0) 618 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:02.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:01 smithi097 bash[24958]: audit 2023-11-17T06:57:00.925666+0000 mon.smithi097 (mon.0) 619 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:57:02.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:01 smithi097 bash[24958]: audit 2023-11-17T06:57:00.932577+0000 mon.smithi097 (mon.0) 620 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:02.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:01 smithi097 bash[24958]: audit 2023-11-17T06:57:00.940223+0000 mon.smithi097 (mon.0) 621 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:57:02.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:01 smithi193 bash[28559]: audit 2023-11-17T06:57:00.658426+0000 mon.smithi097 (mon.0) 617 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:02.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:01 smithi193 bash[28559]: audit 2023-11-17T06:57:00.924827+0000 mon.smithi097 (mon.0) 618 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:02.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:01 smithi193 bash[28559]: audit 2023-11-17T06:57:00.925666+0000 mon.smithi097 (mon.0) 619 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:57:02.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:01 smithi193 bash[28559]: audit 2023-11-17T06:57:00.932577+0000 mon.smithi097 (mon.0) 620 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:02.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:01 smithi193 bash[28559]: audit 2023-11-17T06:57:00.940223+0000 mon.smithi097 (mon.0) 621 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:57:03.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:02 smithi097 bash[24958]: cluster 2023-11-17T06:57:01.420377+0000 mgr.smithi097.swvonl (mgr.14180) 301 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:03.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:02 smithi193 bash[28559]: cluster 2023-11-17T06:57:01.420377+0000 mgr.smithi097.swvonl (mgr.14180) 301 : cluster [DBG] pgmap v280: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager ?:9093,9094 1/1 3s ago 9m count:1 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:crash 2/2 3s ago 9m * 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:grafana ?:3000 1/1 3s ago 9m count:1 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:mgr 2/2 3s ago 9m count:2 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:mon 2/2 3s ago 7m smithi097:172.21.15.97=smithi097;smithi193:172.21.15.193=smithi193;count:2 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter ?:9100 2/2 3s ago 9m * 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:osd.all-available-devices 8 3s ago 7m * 2023-11-17T06:57:03.950 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus ?:9095 1/1 3s ago 9m count:1 2023-11-17T06:57:04.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-17T06:57:05.031 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:04 smithi097 bash[24958]: cluster 2023-11-17T06:57:03.420944+0000 mgr.smithi097.swvonl (mgr.14180) 302 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:05.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:04 smithi193 bash[28559]: cluster 2023-11-17T06:57:03.420944+0000 mgr.smithi097.swvonl (mgr.14180) 302 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:06.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:05 smithi097 bash[24958]: audit 2023-11-17T06:57:03.937079+0000 mgr.smithi097.swvonl (mgr.14180) 303 : audit [DBG] from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:06.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:05 smithi193 bash[28559]: audit 2023-11-17T06:57:03.937079+0000 mgr.smithi097.swvonl (mgr.14180) 303 : audit [DBG] from='client.14482 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:06.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:06 smithi097 bash[24958]: cluster 2023-11-17T06:57:05.421490+0000 mgr.smithi097.swvonl (mgr.14180) 304 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:07.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:06 smithi193 bash[28559]: cluster 2023-11-17T06:57:05.421490+0000 mgr.smithi097.swvonl (mgr.14180) 304 : cluster [DBG] pgmap v282: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:08.449 INFO:teuthology.orchestra.run.smithi097.stdout:HOST ADDR LABELS STATUS 2023-11-17T06:57:08.450 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 172.21.15.97 2023-11-17T06:57:08.450 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 172.21.15.193 2023-11-17T06:57:08.450 INFO:teuthology.orchestra.run.smithi097.stdout:2 hosts in cluster 2023-11-17T06:57:08.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:08 smithi097 bash[24958]: cluster 2023-11-17T06:57:07.422096+0000 mgr.smithi097.swvonl (mgr.14180) 305 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:09.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:08 smithi193 bash[28559]: cluster 2023-11-17T06:57:07.422096+0000 mgr.smithi097.swvonl (mgr.14180) 305 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:09.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-17T06:57:09.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:09 smithi097 bash[24958]: audit 2023-11-17T06:57:08.445492+0000 mgr.smithi097.swvonl (mgr.14180) 306 : audit [DBG] from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:10.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:09 smithi193 bash[28559]: audit 2023-11-17T06:57:08.445492+0000 mgr.smithi097.swvonl (mgr.14180) 306 : audit [DBG] from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:10 smithi097 bash[24958]: cluster 2023-11-17T06:57:09.422619+0000 mgr.smithi097.swvonl (mgr.14180) 307 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:11.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:10 smithi193 bash[28559]: cluster 2023-11-17T06:57:09.422619+0000 mgr.smithi097.swvonl (mgr.14180) 307 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:14.717 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:12 smithi097 bash[24958]: cluster 2023-11-17T06:57:11.423222+0000 mgr.smithi097.swvonl (mgr.14180) 308 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:14.718 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:12 smithi193 bash[28559]: cluster 2023-11-17T06:57:11.423222+0000 mgr.smithi097.swvonl (mgr.14180) 308 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:14.719 INFO:teuthology.orchestra.run.smithi097.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-17T06:57:14.719 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme0n1 ssd INTEL SSDPEDMD400G4_PHFT6204013T400BGN 372G 5m ago LVM detected, locked 2023-11-17T06:57:14.720 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme1n1 ssd Linux_c1e833fed63937b5bcff 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.720 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme2n1 ssd Linux_9dde34b887e30b7fdf21 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.720 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme3n1 ssd Linux_6200f3985392da14bb33 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.720 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme4n1 ssd Linux_e9979a13e8359236589a 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme0n1 ssd INTEL SSDPEDMD400G4_PHFT620500HB400BGN 372G 5m ago LVM detected, locked 2023-11-17T06:57:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme1n1 ssd Linux_8a764a1b7dfaf71f5124 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme2n1 ssd Linux_757d10d3ee3d13b852bd 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme3n1 ssd Linux_6e45eadb96df34fe7a19 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.721 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme4n1 ssd Linux_5ef285d79cb74cea0c4f 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:14.722 INFO:teuthology.run_tasks:Running task vip... 2023-11-17T06:57:14.731 INFO:tasks.vip:Allocating static IPs for each host... 2023-11-17T06:57:14.732 INFO:tasks.vip:peername 172.21.15.97 2023-11-17T06:57:14.737 INFO:tasks.vip:172.21.15.97 in 172.21.0.0/20, pos 3936 2023-11-17T06:57:14.741 INFO:tasks.vip:smithi097.front.sepia.ceph.com static 10.0.15.97, vnet 10.0.0.0/16 2023-11-17T06:57:14.741 INFO:tasks.vip:VIPs are {map(str, vips)} 2023-11-17T06:57:14.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip route ls 2023-11-17T06:57:14.755 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:14 smithi097 bash[24958]: audit 2023-11-17T06:57:13.045873+0000 mgr.smithi097.swvonl (mgr.14180) 309 : audit [DBG] from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:14.755 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:14 smithi097 bash[24958]: cluster 2023-11-17T06:57:13.423844+0000 mgr.smithi097.swvonl (mgr.14180) 310 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:14.757 INFO:teuthology.orchestra.run.smithi097.stdout:default via 172.21.15.254 dev enp3s0f1 2023-11-17T06:57:14.757 INFO:teuthology.orchestra.run.smithi097.stdout:default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.97 metric 100 2023-11-17T06:57:14.758 INFO:teuthology.orchestra.run.smithi097.stdout:172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T06:57:14.758 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.97 2023-11-17T06:57:14.758 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.97 metric 100 2023-11-17T06:57:14.759 INFO:tasks.vip:Configuring 10.0.15.97 on smithi097.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T06:57:14.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr add 10.0.15.97/16 dev enp3s0f1 2023-11-17T06:57:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:14 smithi193 bash[28559]: audit 2023-11-17T06:57:13.045873+0000 mgr.smithi097.swvonl (mgr.14180) 309 : audit [DBG] from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:14 smithi193 bash[28559]: cluster 2023-11-17T06:57:13.423844+0000 mgr.smithi097.swvonl (mgr.14180) 310 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:14.817 INFO:tasks.vip:peername 172.21.15.193 2023-11-17T06:57:14.822 INFO:tasks.vip:172.21.15.193 in 172.21.0.0/20, pos 4032 2023-11-17T06:57:14.829 INFO:tasks.vip:smithi193.front.sepia.ceph.com static 10.0.15.193, vnet 10.0.0.0/16 2023-11-17T06:57:14.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo ip route ls 2023-11-17T06:57:14.845 INFO:teuthology.orchestra.run.smithi193.stdout:default via 172.21.15.254 dev enp3s0f1 2023-11-17T06:57:14.845 INFO:teuthology.orchestra.run.smithi193.stdout:default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.193 metric 100 2023-11-17T06:57:14.845 INFO:teuthology.orchestra.run.smithi193.stdout:172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-11-17T06:57:14.845 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.193 2023-11-17T06:57:14.846 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.193 metric 100 2023-11-17T06:57:14.846 INFO:tasks.vip:Configuring 10.0.15.193 on smithi193.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T06:57:14.847 DEBUG:teuthology.orchestra.run.smithi193:> sudo ip addr add 10.0.15.193/16 dev enp3s0f1 2023-11-17T06:57:14.904 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T06:57:14.915 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T06:57:14.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph orch device ls --refresh' 2023-11-17T06:57:16.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:16 smithi193 bash[28559]: cluster 2023-11-17T06:57:15.424424+0000 mgr.smithi097.swvonl (mgr.14180) 311 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:16.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:16 smithi097 bash[24958]: cluster 2023-11-17T06:57:15.424424+0000 mgr.smithi097.swvonl (mgr.14180) 311 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:18.589 INFO:teuthology.orchestra.run.smithi097.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-17T06:57:18.589 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme0n1 ssd INTEL SSDPEDMD400G4_PHFT6204013T400BGN 372G 5m ago LVM detected, locked 2023-11-17T06:57:18.589 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme1n1 ssd Linux_c1e833fed63937b5bcff 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme2n1 ssd Linux_9dde34b887e30b7fdf21 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme3n1 ssd Linux_6200f3985392da14bb33 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097 /dev/nvme4n1 ssd Linux_e9979a13e8359236589a 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme0n1 ssd INTEL SSDPEDMD400G4_PHFT620500HB400BGN 372G 5m ago LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme1n1 ssd Linux_8a764a1b7dfaf71f5124 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme2n1 ssd Linux_757d10d3ee3d13b852bd 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme3n1 ssd Linux_6e45eadb96df34fe7a19 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.590 INFO:teuthology.orchestra.run.smithi097.stdout:smithi193 /dev/nvme4n1 ssd Linux_5ef285d79cb74cea0c4f 89.4G 5m ago Insufficient space (<10 extents) on vgs, LVM detected, locked 2023-11-17T06:57:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:18 smithi097 bash[24958]: cluster 2023-11-17T06:57:17.424987+0000 mgr.smithi097.swvonl (mgr.14180) 312 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:18 smithi193 bash[28559]: cluster 2023-11-17T06:57:17.424987+0000 mgr.smithi097.swvonl (mgr.14180) 312 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:19.392 INFO:teuthology.run_tasks:Running task vip.exec... 2023-11-17T06:57:19.402 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T06:57:19.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2023-11-17T06:57:19.416 INFO:teuthology.orchestra.run.smithi097.stderr:+ systemctl stop nfs-server 2023-11-17T06:57:19.536 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi193.front.sepia.ceph.com 2023-11-17T06:57:19.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2023-11-17T06:57:19.552 INFO:teuthology.orchestra.run.smithi193.stderr:+ systemctl stop nfs-server 2023-11-17T06:57:19.687 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-17T06:57:19.695 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T06:57:19.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- bash -c 'ceph fs volume create foofs' 2023-11-17T06:57:19.709 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:19 smithi097 bash[24958]: audit 2023-11-17T06:57:18.578781+0000 mgr.smithi097.swvonl (mgr.14180) 313 : audit [DBG] from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:19.709 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:19 smithi097 bash[24958]: audit 2023-11-17T06:57:18.608939+0000 mon.smithi097 (mon.0) 622 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:57:19.709 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:19 smithi097 bash[24958]: audit 2023-11-17T06:57:18.865000+0000 mon.smithi097 (mon.0) 623 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:19 smithi193 bash[28559]: audit 2023-11-17T06:57:18.578781+0000 mgr.smithi097.swvonl (mgr.14180) 313 : audit [DBG] from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:19.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:19 smithi193 bash[28559]: audit 2023-11-17T06:57:18.608939+0000 mon.smithi097 (mon.0) 622 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:57:19.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:19 smithi193 bash[28559]: audit 2023-11-17T06:57:18.865000+0000 mon.smithi097 (mon.0) 623 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:20.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:20 smithi097 bash[24958]: cluster 2023-11-17T06:57:19.425425+0000 mgr.smithi097.swvonl (mgr.14180) 314 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:20 smithi193 bash[28559]: cluster 2023-11-17T06:57:19.425425+0000 mgr.smithi097.swvonl (mgr.14180) 314 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:22.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:22 smithi097 bash[24958]: cluster 2023-11-17T06:57:21.425961+0000 mgr.smithi097.swvonl (mgr.14180) 315 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:22.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:22 smithi193 bash[28559]: cluster 2023-11-17T06:57:21.425961+0000 mgr.smithi097.swvonl (mgr.14180) 315 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:24.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:23 smithi097 bash[24958]: audit 2023-11-17T06:57:23.316081+0000 mgr.smithi097.swvonl (mgr.14180) 316 : audit [DBG] from='client.14498 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:24.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:23 smithi097 bash[24958]: audit 2023-11-17T06:57:23.316882+0000 mon.smithi097 (mon.0) 624 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2023-11-17T06:57:24.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:23 smithi097 bash[24958]: cluster 2023-11-17T06:57:23.426495+0000 mgr.smithi097.swvonl (mgr.14180) 317 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:23 smithi193 bash[28559]: audit 2023-11-17T06:57:23.316081+0000 mgr.smithi097.swvonl (mgr.14180) 316 : audit [DBG] from='client.14498 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:23 smithi193 bash[28559]: audit 2023-11-17T06:57:23.316882+0000 mon.smithi097 (mon.0) 624 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2023-11-17T06:57:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:23 smithi193 bash[28559]: cluster 2023-11-17T06:57:23.426495+0000 mgr.smithi097.swvonl (mgr.14180) 317 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:25.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:24 smithi097 bash[24958]: debug 2023-11-17T06:57:24.843+0000 7f17f2cc5700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T06:57:25.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:24 smithi097 bash[24958]: audit 2023-11-17T06:57:23.826483+0000 mon.smithi097 (mon.0) 625 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2023-11-17T06:57:25.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:24 smithi097 bash[24958]: cluster 2023-11-17T06:57:23.826558+0000 mon.smithi097 (mon.0) 626 : cluster [DBG] osdmap e42: 8 total, 8 up, 8 in 2023-11-17T06:57:25.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:24 smithi097 bash[24958]: audit 2023-11-17T06:57:23.836179+0000 mon.smithi097 (mon.0) 627 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2023-11-17T06:57:25.197 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:24 smithi097 bash[24958]: audit 2023-11-17T06:57:24.277356+0000 mon.smithi097 (mon.0) 628 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:25.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:24 smithi193 bash[28559]: audit 2023-11-17T06:57:23.826483+0000 mon.smithi097 (mon.0) 625 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2023-11-17T06:57:25.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:24 smithi193 bash[28559]: cluster 2023-11-17T06:57:23.826558+0000 mon.smithi097 (mon.0) 626 : cluster [DBG] osdmap e42: 8 total, 8 up, 8 in 2023-11-17T06:57:25.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:24 smithi193 bash[28559]: audit 2023-11-17T06:57:23.836179+0000 mon.smithi097 (mon.0) 627 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2023-11-17T06:57:25.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:24 smithi193 bash[28559]: audit 2023-11-17T06:57:24.277356+0000 mon.smithi097 (mon.0) 628 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:25.850 INFO:teuthology.run_tasks:Running task cephadm.apply... 2023-11-17T06:57:25.861 INFO:tasks.cephadm:Applying spec(s): placement: count: 2 service_id: foo service_type: nfs spec: port: 12049 --- service_id: nfs.foo service_type: ingress spec: backend_service: nfs.foo frontend_port: 2049 monitor_port: 9002 virtual_ip: '10.0.31.97/16' 2023-11-17T06:57:25.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch apply -i - 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: audit 2023-11-17T06:57:24.831484+0000 mon.smithi097 (mon.0) 629 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:24.832126+0000 mon.smithi097 (mon.0) 630 : cluster [DBG] osdmap e43: 8 total, 8 up, 8 in 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: audit 2023-11-17T06:57:24.848131+0000 mon.smithi097 (mon.0) 631 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:24.851345+0000 mon.smithi097 (mon.0) 632 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:24.851480+0000 mon.smithi097 (mon.0) 633 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:24.864241+0000 mon.smithi097 (mon.0) 634 : cluster [DBG] osdmap e44: 8 total, 8 up, 8 in 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: audit 2023-11-17T06:57:24.864356+0000 mon.smithi097 (mon.0) 635 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:24.864425+0000 mon.smithi097 (mon.0) 636 : cluster [DBG] fsmap foofs:0 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cephadm 2023-11-17T06:57:24.868719+0000 mgr.smithi097.swvonl (mgr.14180) 318 : cephadm [INF] Saving service mds.foofs spec with placement count:2 2023-11-17T06:57:26.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: audit 2023-11-17T06:57:24.876601+0000 mon.smithi097 (mon.0) 637 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:26.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: cluster 2023-11-17T06:57:25.426940+0000 mgr.smithi097.swvonl (mgr.14180) 319 : cluster [DBG] pgmap v295: 65 pgs: 26 creating+peering, 6 active+clean, 33 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:26.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:25 smithi097 bash[24958]: audit 2023-11-17T06:57:25.819371+0000 mon.smithi097 (mon.0) 638 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:26.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: audit 2023-11-17T06:57:24.831484+0000 mon.smithi097 (mon.0) 629 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2023-11-17T06:57:26.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:24.832126+0000 mon.smithi097 (mon.0) 630 : cluster [DBG] osdmap e43: 8 total, 8 up, 8 in 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: audit 2023-11-17T06:57:24.848131+0000 mon.smithi097 (mon.0) 631 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:24.851345+0000 mon.smithi097 (mon.0) 632 : cluster [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:24.851480+0000 mon.smithi097 (mon.0) 633 : cluster [WRN] Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:24.864241+0000 mon.smithi097 (mon.0) 634 : cluster [DBG] osdmap e44: 8 total, 8 up, 8 in 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: audit 2023-11-17T06:57:24.864356+0000 mon.smithi097 (mon.0) 635 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:24.864425+0000 mon.smithi097 (mon.0) 636 : cluster [DBG] fsmap foofs:0 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cephadm 2023-11-17T06:57:24.868719+0000 mgr.smithi097.swvonl (mgr.14180) 318 : cephadm [INF] Saving service mds.foofs spec with placement count:2 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: audit 2023-11-17T06:57:24.876601+0000 mon.smithi097 (mon.0) 637 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: cluster 2023-11-17T06:57:25.426940+0000 mgr.smithi097.swvonl (mgr.14180) 319 : cluster [DBG] pgmap v295: 65 pgs: 26 creating+peering, 6 active+clean, 33 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:26.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:25 smithi193 bash[28559]: audit 2023-11-17T06:57:25.819371+0000 mon.smithi097 (mon.0) 638 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:27.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:26 smithi097 bash[24958]: cluster 2023-11-17T06:57:25.869692+0000 mon.smithi097 (mon.0) 639 : cluster [DBG] osdmap e45: 8 total, 8 up, 8 in 2023-11-17T06:57:27.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:26 smithi097 bash[24958]: audit 2023-11-17T06:57:26.513171+0000 mon.smithi097 (mon.0) 640 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:27.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:26 smithi193 bash[28559]: cluster 2023-11-17T06:57:25.869692+0000 mon.smithi097 (mon.0) 639 : cluster [DBG] osdmap e45: 8 total, 8 up, 8 in 2023-11-17T06:57:27.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:26 smithi193 bash[28559]: audit 2023-11-17T06:57:26.513171+0000 mon.smithi097 (mon.0) 640 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:27 smithi097 bash[24958]: cluster 2023-11-17T06:57:26.879712+0000 mon.smithi097 (mon.0) 641 : cluster [DBG] osdmap e46: 8 total, 8 up, 8 in 2023-11-17T06:57:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:27 smithi097 bash[24958]: cluster 2023-11-17T06:57:27.427524+0000 mgr.smithi097.swvonl (mgr.14180) 320 : cluster [DBG] pgmap v298: 65 pgs: 26 creating+peering, 9 active+clean, 30 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:28.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:27 smithi193 bash[28559]: cluster 2023-11-17T06:57:26.879712+0000 mon.smithi097 (mon.0) 641 : cluster [DBG] osdmap e46: 8 total, 8 up, 8 in 2023-11-17T06:57:28.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:27 smithi193 bash[28559]: cluster 2023-11-17T06:57:27.427524+0000 mgr.smithi097.swvonl (mgr.14180) 320 : cluster [DBG] pgmap v298: 65 pgs: 26 creating+peering, 9 active+clean, 30 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:29.611 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled nfs.foo update... 2023-11-17T06:57:29.611 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled ingress.nfs.foo update... 2023-11-17T06:57:30.377 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2023-11-17T06:57:30.386 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2023-11-17T06:57:30.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:57:30.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: cluster 2023-11-17T06:57:29.428187+0000 mgr.smithi097.swvonl (mgr.14180) 321 : cluster [DBG] pgmap v299: 65 pgs: 26 creating+peering, 24 active+clean, 15 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: audit 2023-11-17T06:57:29.590425+0000 mgr.smithi097.swvonl (mgr.14180) 322 : audit [DBG] from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch apply", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: cephadm 2023-11-17T06:57:29.598526+0000 mgr.smithi097.swvonl (mgr.14180) 323 : cephadm [INF] Saving service nfs.foo spec with placement count:2 2023-11-17T06:57:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: audit 2023-11-17T06:57:29.603315+0000 mon.smithi097 (mon.0) 642 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: cephadm 2023-11-17T06:57:29.604176+0000 mgr.smithi097.swvonl (mgr.14180) 324 : cephadm [INF] Saving service ingress.nfs.foo spec with placement count:2 2023-11-17T06:57:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:30 smithi193 bash[28559]: audit 2023-11-17T06:57:29.607766+0000 mon.smithi097 (mon.0) 643 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: cluster 2023-11-17T06:57:29.428187+0000 mgr.smithi097.swvonl (mgr.14180) 321 : cluster [DBG] pgmap v299: 65 pgs: 26 creating+peering, 24 active+clean, 15 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: audit 2023-11-17T06:57:29.590425+0000 mgr.smithi097.swvonl (mgr.14180) 322 : audit [DBG] from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch apply", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: cephadm 2023-11-17T06:57:29.598526+0000 mgr.smithi097.swvonl (mgr.14180) 323 : cephadm [INF] Saving service nfs.foo spec with placement count:2 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: audit 2023-11-17T06:57:29.603315+0000 mon.smithi097 (mon.0) 642 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: cephadm 2023-11-17T06:57:29.604176+0000 mgr.smithi097.swvonl (mgr.14180) 324 : cephadm [INF] Saving service ingress.nfs.foo spec with placement count:2 2023-11-17T06:57:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:30 smithi097 bash[24958]: audit 2023-11-17T06:57:29.607766+0000 mon.smithi097 (mon.0) 643 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:32.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:32 smithi193 bash[28559]: cluster 2023-11-17T06:57:31.428734+0000 mgr.smithi097.swvonl (mgr.14180) 325 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:32.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:32 smithi097 bash[24958]: cluster 2023-11-17T06:57:31.428734+0000 mgr.smithi097.swvonl (mgr.14180) 325 : cluster [DBG] pgmap v300: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:34.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:34 smithi097 bash[24958]: cluster 2023-11-17T06:57:33.429420+0000 mgr.smithi097.swvonl (mgr.14180) 326 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:34.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:34 smithi097 bash[24958]: audit 2023-11-17T06:57:34.464242+0000 mon.smithi097 (mon.0) 644 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:34.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:34 smithi193 bash[28559]: cluster 2023-11-17T06:57:33.429420+0000 mgr.smithi097.swvonl (mgr.14180) 326 : cluster [DBG] pgmap v301: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:34.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:34 smithi193 bash[28559]: audit 2023-11-17T06:57:34.464242+0000 mon.smithi097 (mon.0) 644 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:35.812 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:57:35.812 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:24.104101Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:24.104450Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:24.104693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "ports": [12049], "running": 0, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:24.104920Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:24.105142Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:57:36.630 INFO:tasks.cephadm:nfs.foo has 0/2 2023-11-17T06:57:36.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:36 smithi097 bash[24958]: cluster 2023-11-17T06:57:35.430000+0000 mgr.smithi097.swvonl (mgr.14180) 327 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:36.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:36 smithi193 bash[28559]: cluster 2023-11-17T06:57:35.430000+0000 mgr.smithi097.swvonl (mgr.14180) 327 : cluster [DBG] pgmap v302: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:37.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:57:37.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:37 smithi097 bash[24958]: audit 2023-11-17T06:57:35.799977+0000 mgr.smithi097.swvonl (mgr.14180) 328 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:37.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:37 smithi193 bash[28559]: audit 2023-11-17T06:57:35.799977+0000 mgr.smithi097.swvonl (mgr.14180) 328 : audit [DBG] from='client.14506 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:38.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:38 smithi097 bash[24958]: cluster 2023-11-17T06:57:37.430442+0000 mgr.smithi097.swvonl (mgr.14180) 329 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:38.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:38 smithi193 bash[28559]: cluster 2023-11-17T06:57:37.430442+0000 mgr.smithi097.swvonl (mgr.14180) 329 : cluster [DBG] pgmap v303: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.875798+0000 mon.smithi097 (mon.0) 645 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.876944+0000 mon.smithi097 (mon.0) 646 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.877829+0000 mon.smithi097 (mon.0) 647 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.882810+0000 mon.smithi097 (mon.0) 648 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.887581+0000 mon.smithi097 (mon.0) 649 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.903643+0000 mon.smithi097 (mon.0) 650 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.904692+0000 mon.smithi097 (mon.0) 651 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.uynvbw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T06:57:40.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.908538+0000 mon.smithi097 (mon.0) 652 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.uynvbw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T06:57:40.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: audit 2023-11-17T06:57:38.909330+0000 mon.smithi097 (mon.0) 653 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:40.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: cephadm 2023-11-17T06:57:38.910020+0000 mgr.smithi097.swvonl (mgr.14180) 330 : cephadm [INF] Deploying daemon mds.foofs.smithi193.uynvbw on smithi193 2023-11-17T06:57:40.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:39 smithi097 bash[24958]: cluster 2023-11-17T06:57:39.431141+0000 mgr.smithi097.swvonl (mgr.14180) 331 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.875798+0000 mon.smithi097 (mon.0) 645 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.876944+0000 mon.smithi097 (mon.0) 646 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.877829+0000 mon.smithi097 (mon.0) 647 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.882810+0000 mon.smithi097 (mon.0) 648 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.887581+0000 mon.smithi097 (mon.0) 649 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.903643+0000 mon.smithi097 (mon.0) 650 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.904692+0000 mon.smithi097 (mon.0) 651 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.uynvbw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.908538+0000 mon.smithi097 (mon.0) 652 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi193.uynvbw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T06:57:40.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: audit 2023-11-17T06:57:38.909330+0000 mon.smithi097 (mon.0) 653 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:40.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: cephadm 2023-11-17T06:57:38.910020+0000 mgr.smithi097.swvonl (mgr.14180) 330 : cephadm [INF] Deploying daemon mds.foofs.smithi193.uynvbw on smithi193 2023-11-17T06:57:40.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:39 smithi193 bash[28559]: cluster 2023-11-17T06:57:39.431141+0000 mgr.smithi097.swvonl (mgr.14180) 331 : cluster [DBG] pgmap v304: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:41.636 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:57:41.636 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:24.104101Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:24.104450Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:24.104693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "ports": [12049], "running": 0, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:24.104920Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:24.105142Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:57:42.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: cluster 2023-11-17T06:57:41.431778+0000 mgr.smithi097.swvonl (mgr.14180) 332 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: audit 2023-11-17T06:57:41.502679+0000 mon.smithi097 (mon.0) 654 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: audit 2023-11-17T06:57:41.504519+0000 mon.smithi097 (mon.0) 655 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi097.vbcetq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: audit 2023-11-17T06:57:41.510347+0000 mon.smithi097 (mon.0) 656 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi097.vbcetq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: audit 2023-11-17T06:57:41.513434+0000 mon.smithi097 (mon.0) 657 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: cephadm 2023-11-17T06:57:41.515017+0000 mgr.smithi097.swvonl (mgr.14180) 333 : cephadm [INF] Deploying daemon mds.foofs.smithi097.vbcetq on smithi097 2023-11-17T06:57:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:42 smithi097 bash[24958]: audit 2023-11-17T06:57:41.627419+0000 mgr.smithi097.swvonl (mgr.14180) 334 : audit [DBG] from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: cluster 2023-11-17T06:57:41.431778+0000 mgr.smithi097.swvonl (mgr.14180) 332 : cluster [DBG] pgmap v305: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: audit 2023-11-17T06:57:41.502679+0000 mon.smithi097 (mon.0) 654 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: audit 2023-11-17T06:57:41.504519+0000 mon.smithi097 (mon.0) 655 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi097.vbcetq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-17T06:57:42.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: audit 2023-11-17T06:57:41.510347+0000 mon.smithi097 (mon.0) 656 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi097.vbcetq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-17T06:57:42.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: audit 2023-11-17T06:57:41.513434+0000 mon.smithi097 (mon.0) 657 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:42.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: cephadm 2023-11-17T06:57:41.515017+0000 mgr.smithi097.swvonl (mgr.14180) 333 : cephadm [INF] Deploying daemon mds.foofs.smithi097.vbcetq on smithi097 2023-11-17T06:57:42.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:42 smithi193 bash[28559]: audit 2023-11-17T06:57:41.627419+0000 mgr.smithi097.swvonl (mgr.14180) 334 : audit [DBG] from='client.14510 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:42.864 INFO:tasks.cephadm:nfs.foo has 0/2 2023-11-17T06:57:43.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:57:44.770 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.432455+0000 mgr.smithi097.swvonl (mgr.14180) 335 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506506+0000 mon.smithi097 (mon.0) 658 : cluster [DBG] mds.? [v2:172.21.15.193:6832/4025660031,v1:172.21.15.193:6833/4025660031] up:boot 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506573+0000 mon.smithi097 (mon.0) 659 : cluster [INF] daemon mds.foofs.smithi193.uynvbw assigned to filesystem foofs as rank 0 (now has 1 ranks) 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506644+0000 mon.smithi097 (mon.0) 660 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506664+0000 mon.smithi097 (mon.0) 661 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506685+0000 mon.smithi097 (mon.0) 662 : cluster [INF] Cluster is now healthy 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.506922+0000 mon.smithi097 (mon.0) 663 : cluster [DBG] fsmap foofs:0 1 up:standby 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: audit 2023-11-17T06:57:43.507190+0000 mon.smithi097 (mon.0) 664 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.uynvbw"}]: dispatch 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.514600+0000 mon.smithi097 (mon.0) 665 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:creating} 2023-11-17T06:57:44.771 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:44 smithi097 bash[24958]: cluster 2023-11-17T06:57:43.531884+0000 mon.smithi097 (mon.0) 666 : cluster [INF] daemon mds.foofs.smithi193.uynvbw is now active in filesystem foofs as rank 0 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.432455+0000 mgr.smithi097.swvonl (mgr.14180) 335 : cluster [DBG] pgmap v306: 65 pgs: 65 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506506+0000 mon.smithi097 (mon.0) 658 : cluster [DBG] mds.? [v2:172.21.15.193:6832/4025660031,v1:172.21.15.193:6833/4025660031] up:boot 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506573+0000 mon.smithi097 (mon.0) 659 : cluster [INF] daemon mds.foofs.smithi193.uynvbw assigned to filesystem foofs as rank 0 (now has 1 ranks) 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506644+0000 mon.smithi097 (mon.0) 660 : cluster [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506664+0000 mon.smithi097 (mon.0) 661 : cluster [INF] Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506685+0000 mon.smithi097 (mon.0) 662 : cluster [INF] Cluster is now healthy 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.506922+0000 mon.smithi097 (mon.0) 663 : cluster [DBG] fsmap foofs:0 1 up:standby 2023-11-17T06:57:44.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: audit 2023-11-17T06:57:43.507190+0000 mon.smithi097 (mon.0) 664 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi193.uynvbw"}]: dispatch 2023-11-17T06:57:44.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.514600+0000 mon.smithi097 (mon.0) 665 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:creating} 2023-11-17T06:57:44.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:44 smithi193 bash[28559]: cluster 2023-11-17T06:57:43.531884+0000 mon.smithi097 (mon.0) 666 : cluster [INF] daemon mds.foofs.smithi193.uynvbw is now active in filesystem foofs as rank 0 2023-11-17T06:57:45.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:45 smithi193 bash[28559]: cluster 2023-11-17T06:57:44.513819+0000 mon.smithi097 (mon.0) 667 : cluster [DBG] mds.? [v2:172.21.15.193:6832/4025660031,v1:172.21.15.193:6833/4025660031] up:active 2023-11-17T06:57:45.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:45 smithi193 bash[28559]: cluster 2023-11-17T06:57:44.513876+0000 mon.smithi097 (mon.0) 668 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 2023-11-17T06:57:45.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:45 smithi193 bash[28559]: audit 2023-11-17T06:57:45.049701+0000 mon.smithi097 (mon.0) 669 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:45.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:45 smithi193 bash[28559]: audit 2023-11-17T06:57:45.055831+0000 mon.smithi097 (mon.0) 670 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:45.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:45 smithi097 bash[24958]: cluster 2023-11-17T06:57:44.513819+0000 mon.smithi097 (mon.0) 667 : cluster [DBG] mds.? [v2:172.21.15.193:6832/4025660031,v1:172.21.15.193:6833/4025660031] up:active 2023-11-17T06:57:45.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:45 smithi097 bash[24958]: cluster 2023-11-17T06:57:44.513876+0000 mon.smithi097 (mon.0) 668 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 2023-11-17T06:57:45.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:45 smithi097 bash[24958]: audit 2023-11-17T06:57:45.049701+0000 mon.smithi097 (mon.0) 669 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:45.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:45 smithi097 bash[24958]: audit 2023-11-17T06:57:45.055831+0000 mon.smithi097 (mon.0) 670 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:46 smithi097 bash[24958]: audit 2023-11-17T06:57:45.077906+0000 mon.smithi097 (mon.0) 671 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2023-11-17T06:57:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:46 smithi097 bash[24958]: cluster 2023-11-17T06:57:45.433285+0000 mgr.smithi097.swvonl (mgr.14180) 336 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:46 smithi097 bash[24958]: audit 2023-11-17T06:57:46.062400+0000 mon.smithi097 (mon.0) 672 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2023-11-17T06:57:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:46 smithi097 bash[24958]: cluster 2023-11-17T06:57:46.062492+0000 mon.smithi097 (mon.0) 673 : cluster [DBG] osdmap e47: 8 total, 8 up, 8 in 2023-11-17T06:57:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:46 smithi097 bash[24958]: audit 2023-11-17T06:57:46.064258+0000 mon.smithi097 (mon.0) 674 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2023-11-17T06:57:46.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:46 smithi193 bash[28559]: audit 2023-11-17T06:57:45.077906+0000 mon.smithi097 (mon.0) 671 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2023-11-17T06:57:46.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:46 smithi193 bash[28559]: cluster 2023-11-17T06:57:45.433285+0000 mgr.smithi097.swvonl (mgr.14180) 336 : cluster [DBG] pgmap v307: 65 pgs: 65 active+clean; 2.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:57:46.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:46 smithi193 bash[28559]: audit 2023-11-17T06:57:46.062400+0000 mon.smithi097 (mon.0) 672 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2023-11-17T06:57:46.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:46 smithi193 bash[28559]: cluster 2023-11-17T06:57:46.062492+0000 mon.smithi097 (mon.0) 673 : cluster [DBG] osdmap e47: 8 total, 8 up, 8 in 2023-11-17T06:57:46.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:46 smithi193 bash[28559]: audit 2023-11-17T06:57:46.064258+0000 mon.smithi097 (mon.0) 674 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2023-11-17T06:57:48.524 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.065452+0000 mon.smithi097 (mon.0) 675 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2023-11-17T06:57:48.524 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cluster 2023-11-17T06:57:47.065714+0000 mon.smithi097 (mon.0) 676 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2023-11-17T06:57:48.524 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cephadm 2023-11-17T06:57:47.074308+0000 mgr.smithi097.swvonl (mgr.14180) 337 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi097.friibb 2023-11-17T06:57:48.524 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.075112+0000 mon.smithi097 (mon.0) 677 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.084273+0000 mon.smithi097 (mon.0) 678 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cephadm 2023-11-17T06:57:47.086899+0000 mgr.smithi097.swvonl (mgr.14180) 338 : cephadm [INF] Ensuring nfs.foo.0 is in the ganesha grace table 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.087879+0000 mon.smithi097 (mon.0) 679 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.093450+0000 mon.smithi097 (mon.0) 680 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.095069+0000 mon.smithi097 (mon.0) 681 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.147846+0000 mon.smithi097 (mon.0) 682 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.151335+0000 mon.smithi097 (mon.0) 683 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cephadm 2023-11-17T06:57:47.230214+0000 mgr.smithi097.swvonl (mgr.14180) 339 : cephadm [INF] Creating rados config object: conf-nfs.foo 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cephadm 2023-11-17T06:57:47.293125+0000 mgr.smithi097.swvonl (mgr.14180) 340 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi097.friibb-rgw 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.293938+0000 mon.smithi097 (mon.0) 684 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.300447+0000 mon.smithi097 (mon.0) 685 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: audit 2023-11-17T06:57:47.307233+0000 mon.smithi097 (mon.0) 686 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:48.525 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cephadm 2023-11-17T06:57:47.308811+0000 mgr.smithi097.swvonl (mgr.14180) 341 : cephadm [INF] Deploying daemon nfs.foo.0.0.smithi097.friibb on smithi097 2023-11-17T06:57:48.526 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:48 smithi193 bash[28559]: cluster 2023-11-17T06:57:47.433698+0000 mgr.smithi097.swvonl (mgr.14180) 342 : cluster [DBG] pgmap v310: 97 pgs: 4 creating+peering, 28 unknown, 65 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2023-11-17T06:57:48.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.065452+0000 mon.smithi097 (mon.0) 675 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2023-11-17T06:57:48.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cluster 2023-11-17T06:57:47.065714+0000 mon.smithi097 (mon.0) 676 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2023-11-17T06:57:48.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cephadm 2023-11-17T06:57:47.074308+0000 mgr.smithi097.swvonl (mgr.14180) 337 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi097.friibb 2023-11-17T06:57:48.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.075112+0000 mon.smithi097 (mon.0) 677 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.084273+0000 mon.smithi097 (mon.0) 678 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cephadm 2023-11-17T06:57:47.086899+0000 mgr.smithi097.swvonl (mgr.14180) 338 : cephadm [INF] Ensuring nfs.foo.0 is in the ganesha grace table 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.087879+0000 mon.smithi097 (mon.0) 679 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.093450+0000 mon.smithi097 (mon.0) 680 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.095069+0000 mon.smithi097 (mon.0) 681 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.147846+0000 mon.smithi097 (mon.0) 682 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.151335+0000 mon.smithi097 (mon.0) 683 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cephadm 2023-11-17T06:57:47.230214+0000 mgr.smithi097.swvonl (mgr.14180) 339 : cephadm [INF] Creating rados config object: conf-nfs.foo 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cephadm 2023-11-17T06:57:47.293125+0000 mgr.smithi097.swvonl (mgr.14180) 340 : cephadm [INF] Creating key for client.nfs.foo.0.0.smithi097.friibb-rgw 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.293938+0000 mon.smithi097 (mon.0) 684 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.300447+0000 mon.smithi097 (mon.0) 685 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi097.friibb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T06:57:48.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: audit 2023-11-17T06:57:47.307233+0000 mon.smithi097 (mon.0) 686 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:48.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cephadm 2023-11-17T06:57:47.308811+0000 mgr.smithi097.swvonl (mgr.14180) 341 : cephadm [INF] Deploying daemon nfs.foo.0.0.smithi097.friibb on smithi097 2023-11-17T06:57:48.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:48 smithi097 bash[24958]: cluster 2023-11-17T06:57:47.433698+0000 mgr.smithi097.swvonl (mgr.14180) 342 : cluster [DBG] pgmap v310: 97 pgs: 4 creating+peering, 28 unknown, 65 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2023-11-17T06:57:49.427 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: cluster 2023-11-17T06:57:48.070284+0000 mon.smithi097 (mon.0) 687 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-11-17T06:57:49.427 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: cluster 2023-11-17T06:57:48.307234+0000 mon.smithi097 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.97:6834/3770736920,v1:172.21.15.97:6835/3770736920] up:boot 2023-11-17T06:57:49.428 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: cluster 2023-11-17T06:57:48.307561+0000 mon.smithi097 (mon.0) 689 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 1 up:standby 2023-11-17T06:57:49.428 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: audit 2023-11-17T06:57:48.308169+0000 mon.smithi097 (mon.0) 690 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi097.vbcetq"}]: dispatch 2023-11-17T06:57:49.428 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: cluster 2023-11-17T06:57:48.312151+0000 mon.smithi097 (mon.0) 691 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 1 up:standby 2023-11-17T06:57:49.428 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:49 smithi097 bash[24958]: audit 2023-11-17T06:57:48.363138+0000 mon.smithi097 (mon.0) 692 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:49.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: cluster 2023-11-17T06:57:48.070284+0000 mon.smithi097 (mon.0) 687 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2023-11-17T06:57:49.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: cluster 2023-11-17T06:57:48.307234+0000 mon.smithi097 (mon.0) 688 : cluster [DBG] mds.? [v2:172.21.15.97:6834/3770736920,v1:172.21.15.97:6835/3770736920] up:boot 2023-11-17T06:57:49.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: cluster 2023-11-17T06:57:48.307561+0000 mon.smithi097 (mon.0) 689 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 1 up:standby 2023-11-17T06:57:49.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: audit 2023-11-17T06:57:48.308169+0000 mon.smithi097 (mon.0) 690 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi097.vbcetq"}]: dispatch 2023-11-17T06:57:49.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: cluster 2023-11-17T06:57:48.312151+0000 mon.smithi097 (mon.0) 691 : cluster [DBG] fsmap foofs:1 {0=foofs.smithi193.uynvbw=up:active} 1 up:standby 2023-11-17T06:57:49.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:49 smithi193 bash[28559]: audit 2023-11-17T06:57:48.363138+0000 mon.smithi097 (mon.0) 692 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:49.623 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:57:49.623 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:24.104101Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:24.104450Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:24.104693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "ports": [12049], "running": 0, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:24.104920Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:24.105142Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:57:50.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:50 smithi097 bash[24958]: cluster 2023-11-17T06:57:49.434402+0000 mgr.smithi097.swvonl (mgr.14180) 343 : cluster [DBG] pgmap v312: 97 pgs: 4 creating+peering, 20 unknown, 73 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 1 op/s 2023-11-17T06:57:50.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:50 smithi097 bash[24958]: audit 2023-11-17T06:57:49.610380+0000 mgr.smithi097.swvonl (mgr.14180) 344 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:50 smithi193 bash[28559]: cluster 2023-11-17T06:57:49.434402+0000 mgr.smithi097.swvonl (mgr.14180) 343 : cluster [DBG] pgmap v312: 97 pgs: 4 creating+peering, 20 unknown, 73 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 1 op/s 2023-11-17T06:57:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:50 smithi193 bash[28559]: audit 2023-11-17T06:57:49.610380+0000 mgr.smithi097.swvonl (mgr.14180) 344 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:51.121 INFO:tasks.cephadm:nfs.foo has 0/2 2023-11-17T06:57:52.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.223097+0000 mon.smithi097 (mon.0) 693 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cephadm 2023-11-17T06:57:51.224647+0000 mgr.smithi097.swvonl (mgr.14180) 345 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi193.renpbu 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.224976+0000 mon.smithi097 (mon.0) 694 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.228808+0000 mon.smithi097 (mon.0) 695 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cephadm 2023-11-17T06:57:51.229650+0000 mgr.smithi097.swvonl (mgr.14180) 346 : cephadm [INF] Ensuring nfs.foo.1 is in the ganesha grace table 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.230135+0000 mon.smithi097 (mon.0) 696 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.234738+0000 mon.smithi097 (mon.0) 697 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.236456+0000 mon.smithi097 (mon.0) 698 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.294478+0000 mon.smithi097 (mon.0) 699 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T06:57:52.404 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.299733+0000 mon.smithi097 (mon.0) 700 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cephadm 2023-11-17T06:57:51.355612+0000 mgr.smithi097.swvonl (mgr.14180) 347 : cephadm [INF] Rados config object exists: conf-nfs.foo 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cephadm 2023-11-17T06:57:51.355784+0000 mgr.smithi097.swvonl (mgr.14180) 348 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi193.renpbu-rgw 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.356325+0000 mon.smithi097 (mon.0) 701 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.360627+0000 mon.smithi097 (mon.0) 702 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: audit 2023-11-17T06:57:51.362284+0000 mon.smithi097 (mon.0) 703 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cephadm 2023-11-17T06:57:51.363452+0000 mgr.smithi097.swvonl (mgr.14180) 349 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi193.renpbu on smithi193 2023-11-17T06:57:52.405 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:52 smithi097 bash[24958]: cluster 2023-11-17T06:57:51.435732+0000 mgr.smithi097.swvonl (mgr.14180) 350 : cluster [DBG] pgmap v313: 97 pgs: 4 creating+peering, 93 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 511 B/s wr, 1 op/s 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.223097+0000 mon.smithi097 (mon.0) 693 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cephadm 2023-11-17T06:57:51.224647+0000 mgr.smithi097.swvonl (mgr.14180) 345 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi193.renpbu 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.224976+0000 mon.smithi097 (mon.0) 694 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.228808+0000 mon.smithi097 (mon.0) 695 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cephadm 2023-11-17T06:57:51.229650+0000 mgr.smithi097.swvonl (mgr.14180) 346 : cephadm [INF] Ensuring nfs.foo.1 is in the ganesha grace table 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.230135+0000 mon.smithi097 (mon.0) 696 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.234738+0000 mon.smithi097 (mon.0) 697 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.236456+0000 mon.smithi097 (mon.0) 698 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:52.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.294478+0000 mon.smithi097 (mon.0) 699 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.299733+0000 mon.smithi097 (mon.0) 700 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cephadm 2023-11-17T06:57:51.355612+0000 mgr.smithi097.swvonl (mgr.14180) 347 : cephadm [INF] Rados config object exists: conf-nfs.foo 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cephadm 2023-11-17T06:57:51.355784+0000 mgr.smithi097.swvonl (mgr.14180) 348 : cephadm [INF] Creating key for client.nfs.foo.1.0.smithi193.renpbu-rgw 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.356325+0000 mon.smithi097 (mon.0) 701 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.360627+0000 mon.smithi097 (mon.0) 702 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi193.renpbu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: audit 2023-11-17T06:57:51.362284+0000 mon.smithi097 (mon.0) 703 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cephadm 2023-11-17T06:57:51.363452+0000 mgr.smithi097.swvonl (mgr.14180) 349 : cephadm [INF] Deploying daemon nfs.foo.1.0.smithi193.renpbu on smithi193 2023-11-17T06:57:52.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:52 smithi193 bash[28559]: cluster 2023-11-17T06:57:51.435732+0000 mgr.smithi097.swvonl (mgr.14180) 350 : cluster [DBG] pgmap v313: 97 pgs: 4 creating+peering, 93 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 511 B/s wr, 1 op/s 2023-11-17T06:57:54.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:54 smithi097 bash[24958]: cluster 2023-11-17T06:57:53.436574+0000 mgr.smithi097.swvonl (mgr.14180) 351 : cluster [DBG] pgmap v314: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 138 B/s rd, 416 B/s wr, 1 op/s 2023-11-17T06:57:54.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:54 smithi097 bash[24958]: audit 2023-11-17T06:57:54.010385+0000 mon.smithi097 (mon.0) 704 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:54.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:54 smithi097 bash[24958]: audit 2023-11-17T06:57:54.019388+0000 mon.smithi097 (mon.0) 705 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:54.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:54 smithi193 bash[28559]: cluster 2023-11-17T06:57:53.436574+0000 mgr.smithi097.swvonl (mgr.14180) 351 : cluster [DBG] pgmap v314: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 138 B/s rd, 416 B/s wr, 1 op/s 2023-11-17T06:57:54.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:54 smithi193 bash[28559]: audit 2023-11-17T06:57:54.010385+0000 mon.smithi097 (mon.0) 704 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:54.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:54 smithi193 bash[28559]: audit 2023-11-17T06:57:54.019388+0000 mon.smithi097 (mon.0) 705 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:55.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:55 smithi193 bash[28559]: cephadm 2023-11-17T06:57:54.032702+0000 mgr.smithi097.swvonl (mgr.14180) 352 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.wuqmew on smithi193 2023-11-17T06:57:55.707 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:55 smithi097 bash[24958]: cephadm 2023-11-17T06:57:54.032702+0000 mgr.smithi097.swvonl (mgr.14180) 352 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.wuqmew on smithi193 2023-11-17T06:57:56.059 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:57:56.059 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:24.104101Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:24.104450Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:24.104693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "ports": [12049], "running": 0, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:24.104920Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:24.105142Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:57:56.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:56 smithi097 bash[24958]: cluster 2023-11-17T06:57:55.437887+0000 mgr.smithi097.swvonl (mgr.14180) 353 : cluster [DBG] pgmap v315: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2023-11-17T06:57:56.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:56 smithi193 bash[28559]: cluster 2023-11-17T06:57:55.437887+0000 mgr.smithi097.swvonl (mgr.14180) 353 : cluster [DBG] pgmap v315: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2023-11-17T06:57:56.963 INFO:tasks.cephadm:nfs.foo has 0/2 2023-11-17T06:57:57.682 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:57 smithi193 bash[28559]: audit 2023-11-17T06:57:56.045139+0000 mgr.smithi097.swvonl (mgr.14180) 354 : audit [DBG] from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:57.682 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:57 smithi193 bash[28559]: audit 2023-11-17T06:57:56.511461+0000 mon.smithi097 (mon.0) 706 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:57.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:57 smithi097 bash[24958]: audit 2023-11-17T06:57:56.045139+0000 mgr.smithi097.swvonl (mgr.14180) 354 : audit [DBG] from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:57:57.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:57 smithi097 bash[24958]: audit 2023-11-17T06:57:56.511461+0000 mon.smithi097 (mon.0) 706 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:57.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:57:58.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:58 smithi097 bash[24958]: cluster 2023-11-17T06:57:57.438714+0000 mgr.smithi097.swvonl (mgr.14180) 355 : cluster [DBG] pgmap v316: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-17T06:57:58.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:58 smithi097 bash[24958]: audit 2023-11-17T06:57:57.756108+0000 mon.smithi097 (mon.0) 707 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.wuqmew"}]: dispatch 2023-11-17T06:57:58.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:58 smithi097 bash[24958]: audit 2023-11-17T06:57:58.390061+0000 mon.smithi097 (mon.0) 708 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:58 smithi193 bash[28559]: cluster 2023-11-17T06:57:57.438714+0000 mgr.smithi097.swvonl (mgr.14180) 355 : cluster [DBG] pgmap v316: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 2 op/s 2023-11-17T06:57:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:58 smithi193 bash[28559]: audit 2023-11-17T06:57:57.756108+0000 mon.smithi097 (mon.0) 707 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.wuqmew"}]: dispatch 2023-11-17T06:57:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:58 smithi193 bash[28559]: audit 2023-11-17T06:57:58.390061+0000 mon.smithi097 (mon.0) 708 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:57:59.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: cephadm 2023-11-17T06:57:57.755042+0000 mgr.smithi097.swvonl (mgr.14180) 356 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: cephadm 2023-11-17T06:57:57.755546+0000 mgr.smithi097.swvonl (mgr.14180) 357 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: cephadm 2023-11-17T06:57:57.757703+0000 mgr.smithi097.swvonl (mgr.14180) 358 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.wuqmew on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:57:59 smithi097 bash[24958]: cephadm 2023-11-17T06:57:57.761690+0000 mgr.smithi097.swvonl (mgr.14180) 359 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pvrdhk on smithi097 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: cephadm 2023-11-17T06:57:57.755042+0000 mgr.smithi097.swvonl (mgr.14180) 356 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: cephadm 2023-11-17T06:57:57.755546+0000 mgr.smithi097.swvonl (mgr.14180) 357 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: cephadm 2023-11-17T06:57:57.757703+0000 mgr.smithi097.swvonl (mgr.14180) 358 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.wuqmew on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew 2023-11-17T06:57:59.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wuqmew ... 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:57:59.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:57:59 smithi193 bash[28559]: cephadm 2023-11-17T06:57:57.761690+0000 mgr.smithi097.swvonl (mgr.14180) 359 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pvrdhk on smithi097 2023-11-17T06:58:00.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:00 smithi097 bash[24958]: cluster 2023-11-17T06:57:59.439530+0000 mgr.smithi097.swvonl (mgr.14180) 360 : cluster [DBG] pgmap v317: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 990 B/s rd, 900 B/s wr, 2 op/s 2023-11-17T06:58:00.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:00 smithi193 bash[28559]: cluster 2023-11-17T06:57:59.439530+0000 mgr.smithi097.swvonl (mgr.14180) 360 : cluster [DBG] pgmap v317: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 990 B/s rd, 900 B/s wr, 2 op/s 2023-11-17T06:58:01.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:01.743 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:24.104101Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\"", "2023-11-17T06:57:57.757460Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wuqmew on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\nDeploy daemon haproxy.nfs.foo.smithi193.wuqmew ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.565682Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pvrdhk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\nDeploy daemon haproxy.nfs.foo.smithi097.pvrdhk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.574737Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.plhuai on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:01.577189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:24.104450Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:24.104693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "ports": [12049], "running": 0, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:24.104920Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:24.105142Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:02.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cluster 2023-11-17T06:58:01.440234+0000 mgr.smithi097.swvonl (mgr.14180) 361 : cluster [DBG] pgmap v318: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-11-17T06:58:02.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cephadm 2023-11-17T06:58:01.563604+0000 mgr.smithi097.swvonl (mgr.14180) 362 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:58:02.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:02.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cephadm 2023-11-17T06:58:01.564062+0000 mgr.smithi097.swvonl (mgr.14180) 363 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: audit 2023-11-17T06:58:01.564569+0000 mon.smithi097 (mon.0) 709 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pvrdhk"}]: dispatch 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cephadm 2023-11-17T06:58:01.565925+0000 mgr.smithi097.swvonl (mgr.14180) 364 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pvrdhk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:02.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: audit 2023-11-17T06:58:01.574457+0000 mon.smithi097 (mon.0) 710 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cephadm 2023-11-17T06:58:01.574907+0000 mgr.smithi097.swvonl (mgr.14180) 365 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.plhuai on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cephadm 2023-11-17T06:58:01.577308+0000 mgr.smithi097.swvonl (mgr.14180) 366 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: cluster 2023-11-17T06:58:01.578455+0000 mgr.smithi097.swvonl (mgr.14180) 367 : cluster [DBG] pgmap v319: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: audit 2023-11-17T06:58:01.578824+0000 mon.smithi097 (mon.0) 711 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:58:02.305 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:02 smithi097 bash[24958]: audit 2023-11-17T06:58:01.729246+0000 mgr.smithi097.swvonl (mgr.14180) 368 : audit [DBG] from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:02.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cluster 2023-11-17T06:58:01.440234+0000 mgr.smithi097.swvonl (mgr.14180) 361 : cluster [DBG] pgmap v318: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2023-11-17T06:58:02.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cephadm 2023-11-17T06:58:01.563604+0000 mgr.smithi097.swvonl (mgr.14180) 362 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:02.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:02.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:02.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:02.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:58:02.712 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:02.763 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:58:02.835 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:02.913 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:58:02.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:03.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:03.125 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:03.195 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cephadm 2023-11-17T06:58:01.564062+0000 mgr.smithi097.swvonl (mgr.14180) 363 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: audit 2023-11-17T06:58:01.564569+0000 mon.smithi097 (mon.0) 709 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pvrdhk"}]: dispatch 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cephadm 2023-11-17T06:58:01.565925+0000 mgr.smithi097.swvonl (mgr.14180) 364 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pvrdhk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:03.268 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pvrdhk ... 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: audit 2023-11-17T06:58:01.574457+0000 mon.smithi097 (mon.0) 710 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cephadm 2023-11-17T06:58:01.574907+0000 mgr.smithi097.swvonl (mgr.14180) 365 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.plhuai on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cephadm 2023-11-17T06:58:01.577308+0000 mgr.smithi097.swvonl (mgr.14180) 366 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:03.269 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: cluster 2023-11-17T06:58:01.578455+0000 mgr.smithi097.swvonl (mgr.14180) 367 : cluster [DBG] pgmap v319: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-11-17T06:58:03.270 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: audit 2023-11-17T06:58:01.578824+0000 mon.smithi097 (mon.0) 711 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:58:03.270 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:02 smithi193 bash[28559]: audit 2023-11-17T06:58:01.729246+0000 mgr.smithi097.swvonl (mgr.14180) 368 : audit [DBG] from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:03.270 INFO:tasks.cephadm:nfs.foo has 0/2 2023-11-17T06:58:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:03 smithi097 bash[24958]: cluster 2023-11-17T06:58:02.199413+0000 mon.smithi097 (mon.0) 712 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:03 smithi193 bash[28559]: cluster 2023-11-17T06:58:02.199413+0000 mon.smithi097 (mon.0) 712 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:04.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:04.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:04 smithi097 bash[24958]: audit 2023-11-17T06:58:03.400434+0000 mon.smithi097 (mon.0) 713 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:04.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:04 smithi097 bash[24958]: cluster 2023-11-17T06:58:03.579348+0000 mgr.smithi097.swvonl (mgr.14180) 369 : cluster [DBG] pgmap v320: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-11-17T06:58:04.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:04 smithi193 bash[28559]: audit 2023-11-17T06:58:03.400434+0000 mon.smithi097 (mon.0) 713 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:04.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:04 smithi193 bash[28559]: cluster 2023-11-17T06:58:03.579348+0000 mgr.smithi097.swvonl (mgr.14180) 369 : cluster [DBG] pgmap v320: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2023-11-17T06:58:07.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:06 smithi097 bash[24958]: cluster 2023-11-17T06:58:05.580576+0000 mgr.smithi097.swvonl (mgr.14180) 370 : cluster [DBG] pgmap v321: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 908 B/s wr, 3 op/s 2023-11-17T06:58:07.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:06 smithi193 bash[28559]: cluster 2023-11-17T06:58:05.580576+0000 mgr.smithi097.swvonl (mgr.14180) 370 : cluster [DBG] pgmap v321: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 908 B/s wr, 3 op/s 2023-11-17T06:58:07.737 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:07.738 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:57:25.810002Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:57:25.810126Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:57:25.810260Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:29.608049Z service:ingress.nfs.foo [INFO] \"service was created\"", "2023-11-17T06:57:57.757460Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wuqmew on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\nDeploy daemon haproxy.nfs.foo.smithi193.wuqmew ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.565682Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pvrdhk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\nDeploy daemon haproxy.nfs.foo.smithi097.pvrdhk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.574737Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.plhuai on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:01.577189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:07.288502Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:57:25.809875Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:57:25.809661Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:07.288620Z", "ports": [12049], "running": 1, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:57:25.810387Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:57:25.810624Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:57:25.810506Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:08.544 INFO:tasks.cephadm:nfs.foo has 1/2 2023-11-17T06:58:08.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:08 smithi193 bash[28559]: audit 2023-11-17T06:58:07.296999+0000 mon.smithi097 (mon.0) 714 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:08.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:08 smithi193 bash[28559]: cluster 2023-11-17T06:58:07.581212+0000 mgr.smithi097.swvonl (mgr.14180) 371 : cluster [DBG] pgmap v322: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 807 B/s wr, 3 op/s 2023-11-17T06:58:08.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:08 smithi193 bash[28559]: audit 2023-11-17T06:58:07.726367+0000 mgr.smithi097.swvonl (mgr.14180) 372 : audit [DBG] from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:08.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:08 smithi193 bash[28559]: audit 2023-11-17T06:58:08.235471+0000 mon.smithi097 (mon.0) 715 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:08.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:08 smithi097 bash[24958]: audit 2023-11-17T06:58:07.296999+0000 mon.smithi097 (mon.0) 714 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:08.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:08 smithi097 bash[24958]: cluster 2023-11-17T06:58:07.581212+0000 mgr.smithi097.swvonl (mgr.14180) 371 : cluster [DBG] pgmap v322: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 807 B/s wr, 3 op/s 2023-11-17T06:58:08.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:08 smithi097 bash[24958]: audit 2023-11-17T06:58:07.726367+0000 mgr.smithi097.swvonl (mgr.14180) 372 : audit [DBG] from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:08.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:08 smithi097 bash[24958]: audit 2023-11-17T06:58:08.235471+0000 mon.smithi097 (mon.0) 715 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:09.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:09.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: audit 2023-11-17T06:58:08.492522+0000 mon.smithi097 (mon.0) 716 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:58:09.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: audit 2023-11-17T06:58:08.494156+0000 mon.smithi097 (mon.0) 717 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:58:09.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: audit 2023-11-17T06:58:08.502292+0000 mon.smithi097 (mon.0) 718 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:09.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: cluster 2023-11-17T06:58:08.502917+0000 mgr.smithi097.swvonl (mgr.14180) 373 : cluster [DBG] pgmap v323: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 903 B/s wr, 3 op/s 2023-11-17T06:58:09.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: audit 2023-11-17T06:58:08.509298+0000 mon.smithi097 (mon.0) 719 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:58:09.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: audit 2023-11-17T06:58:08.523675+0000 mon.smithi097 (mon.0) 720 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:09.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:09 smithi097 bash[24958]: cephadm 2023-11-17T06:58:08.527853+0000 mgr.smithi097.swvonl (mgr.14180) 374 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rddnon on smithi193 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: audit 2023-11-17T06:58:08.492522+0000 mon.smithi097 (mon.0) 716 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: audit 2023-11-17T06:58:08.494156+0000 mon.smithi097 (mon.0) 717 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: audit 2023-11-17T06:58:08.502292+0000 mon.smithi097 (mon.0) 718 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: cluster 2023-11-17T06:58:08.502917+0000 mgr.smithi097.swvonl (mgr.14180) 373 : cluster [DBG] pgmap v323: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 903 B/s wr, 3 op/s 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: audit 2023-11-17T06:58:08.509298+0000 mon.smithi097 (mon.0) 719 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: audit 2023-11-17T06:58:08.523675+0000 mon.smithi097 (mon.0) 720 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:09.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:09 smithi193 bash[28559]: cephadm 2023-11-17T06:58:08.527853+0000 mgr.smithi097.swvonl (mgr.14180) 374 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rddnon on smithi193 2023-11-17T06:58:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:10 smithi097 bash[24958]: cluster 2023-11-17T06:58:09.500441+0000 mon.smithi097 (mon.0) 721 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:58:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:10 smithi097 bash[24958]: cluster 2023-11-17T06:58:09.500499+0000 mon.smithi097 (mon.0) 722 : cluster [INF] Cluster is now healthy 2023-11-17T06:58:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:10 smithi193 bash[28559]: cluster 2023-11-17T06:58:09.500441+0000 mon.smithi097 (mon.0) 721 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:58:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:10 smithi193 bash[28559]: cluster 2023-11-17T06:58:09.500499+0000 mon.smithi097 (mon.0) 722 : cluster [INF] Cluster is now healthy 2023-11-17T06:58:11.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:11 smithi097 bash[24958]: cluster 2023-11-17T06:58:10.504053+0000 mgr.smithi097.swvonl (mgr.14180) 375 : cluster [DBG] pgmap v324: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 225 B/s rd, 0 op/s 2023-11-17T06:58:11.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:11 smithi193 bash[28559]: cluster 2023-11-17T06:58:10.504053+0000 mgr.smithi097.swvonl (mgr.14180) 375 : cluster [DBG] pgmap v324: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 225 B/s rd, 0 op/s 2023-11-17T06:58:12.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:12 smithi097 bash[24958]: audit 2023-11-17T06:58:12.407390+0000 mon.smithi097 (mon.0) 723 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rddnon"}]: dispatch 2023-11-17T06:58:12.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:12 smithi193 bash[28559]: audit 2023-11-17T06:58:12.407390+0000 mon.smithi097 (mon.0) 723 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rddnon"}]: dispatch 2023-11-17T06:58:13.278 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:13.278 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:08.224155Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:07.287288Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:08.224372Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:57:57.757460Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wuqmew on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wuqmew\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wuqmew\nDeploy daemon haproxy.nfs.foo.smithi193.wuqmew ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.565682Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pvrdhk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pvrdhk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pvrdhk\nDeploy daemon haproxy.nfs.foo.smithi097.pvrdhk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:01.574737Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.plhuai on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:01.577189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:12.408554Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\nDeploy daemon haproxy.nfs.foo.smithi193.rddnon ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:07.288502Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:07.287533Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:07.287668Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:07.288620Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:07.287845Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:07.288000Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:08.224572Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:13.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: cephadm 2023-11-17T06:58:12.406392+0000 mgr.smithi097.swvonl (mgr.14180) 376 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:13.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: cephadm 2023-11-17T06:58:12.406872+0000 mgr.smithi097.swvonl (mgr.14180) 377 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: cephadm 2023-11-17T06:58:12.408810+0000 mgr.smithi097.swvonl (mgr.14180) 378 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: cephadm 2023-11-17T06:58:12.412430+0000 mgr.smithi097.swvonl (mgr.14180) 379 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vtwats on smithi097 2023-11-17T06:58:13.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:13 smithi097 bash[24958]: cluster 2023-11-17T06:58:12.504496+0000 mgr.smithi097.swvonl (mgr.14180) 380 : cluster [DBG] pgmap v325: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: cephadm 2023-11-17T06:58:12.406392+0000 mgr.smithi097.swvonl (mgr.14180) 376 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: cephadm 2023-11-17T06:58:12.406872+0000 mgr.smithi097.swvonl (mgr.14180) 377 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: cephadm 2023-11-17T06:58:12.408810+0000 mgr.smithi097.swvonl (mgr.14180) 378 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rddnon ... 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: cephadm 2023-11-17T06:58:12.412430+0000 mgr.smithi097.swvonl (mgr.14180) 379 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vtwats on smithi097 2023-11-17T06:58:13.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:13 smithi193 bash[28559]: cluster 2023-11-17T06:58:12.504496+0000 mgr.smithi097.swvonl (mgr.14180) 380 : cluster [DBG] pgmap v325: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:14.118 INFO:tasks.cephadm:nfs.foo has 2/2 2023-11-17T06:58:14.118 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2023-11-17T06:58:14.128 INFO:tasks.cephadm:Waiting for ceph service ingress.nfs.foo to start (timeout 300)... 2023-11-17T06:58:14.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:14 smithi097 bash[24958]: audit 2023-11-17T06:58:13.264303+0000 mgr.smithi097.swvonl (mgr.14180) 381 : audit [DBG] from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:14 smithi193 bash[28559]: audit 2023-11-17T06:58:13.264303+0000 mgr.smithi097.swvonl (mgr.14180) 381 : audit [DBG] from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:15.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:15 smithi193 bash[28559]: cluster 2023-11-17T06:58:14.505303+0000 mgr.smithi097.swvonl (mgr.14180) 382 : cluster [DBG] pgmap v326: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:15.769 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:15 smithi097 bash[24958]: cluster 2023-11-17T06:58:14.505303+0000 mgr.smithi097.swvonl (mgr.14180) 382 : cluster [DBG] pgmap v326: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cluster 2023-11-17T06:58:16.506461+0000 mgr.smithi097.swvonl (mgr.14180) 383 : cluster [DBG] pgmap v327: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cephadm 2023-11-17T06:58:16.607706+0000 mgr.smithi097.swvonl (mgr.14180) 384 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cephadm 2023-11-17T06:58:16.607940+0000 mgr.smithi097.swvonl (mgr.14180) 385 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: audit 2023-11-17T06:58:16.608486+0000 mon.smithi097 (mon.0) 724 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vtwats"}]: dispatch 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cephadm 2023-11-17T06:58:16.609548+0000 mgr.smithi097.swvonl (mgr.14180) 386 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vtwats on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cephadm 2023-11-17T06:58:16.611765+0000 mgr.smithi097.swvonl (mgr.14180) 387 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xfospj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:17.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cephadm 2023-11-17T06:58:16.613286+0000 mgr.smithi097.swvonl (mgr.14180) 388 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:17.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: cluster 2023-11-17T06:58:16.614274+0000 mgr.smithi097.swvonl (mgr.14180) 389 : cluster [DBG] pgmap v328: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2023-11-17T06:58:17.558 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:17 smithi097 bash[24958]: audit 2023-11-17T06:58:16.618549+0000 mon.smithi097 (mon.0) 725 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:58:17.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cluster 2023-11-17T06:58:16.506461+0000 mgr.smithi097.swvonl (mgr.14180) 383 : cluster [DBG] pgmap v327: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2023-11-17T06:58:17.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cephadm 2023-11-17T06:58:16.607706+0000 mgr.smithi097.swvonl (mgr.14180) 384 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cephadm 2023-11-17T06:58:16.607940+0000 mgr.smithi097.swvonl (mgr.14180) 385 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: audit 2023-11-17T06:58:16.608486+0000 mon.smithi097 (mon.0) 724 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vtwats"}]: dispatch 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cephadm 2023-11-17T06:58:16.609548+0000 mgr.smithi097.swvonl (mgr.14180) 386 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vtwats on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vtwats ... 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:17.564 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cephadm 2023-11-17T06:58:16.611765+0000 mgr.smithi097.swvonl (mgr.14180) 387 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xfospj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cephadm 2023-11-17T06:58:16.613286+0000 mgr.smithi097.swvonl (mgr.14180) 388 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: cluster 2023-11-17T06:58:16.614274+0000 mgr.smithi097.swvonl (mgr.14180) 389 : cluster [DBG] pgmap v328: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 226 B/s rd, 0 op/s 2023-11-17T06:58:17.565 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:17 smithi193 bash[28559]: audit 2023-11-17T06:58:16.618549+0000 mon.smithi097 (mon.0) 725 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:58:18.153 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:18.153 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:08.224155Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:07.287288Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:08.224372Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:01.577189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:12.408554Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\nDeploy daemon haproxy.nfs.foo.smithi193.rddnon ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.609373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vtwats on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\nDeploy daemon haproxy.nfs.foo.smithi097.vtwats ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.611669Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xfospj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:07.288502Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:07.287533Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:07.287668Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:07.288620Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:07.287845Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:07.288000Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:08.224572Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:18.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:18 smithi097 bash[24958]: cluster 2023-11-17T06:58:17.204350+0000 mon.smithi097 (mon.0) 726 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:18.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:18 smithi193 bash[28559]: cluster 2023-11-17T06:58:17.204350+0000 mon.smithi097 (mon.0) 726 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:18.948 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:58:19.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:19 smithi097 bash[24958]: audit 2023-11-17T06:58:18.143873+0000 mgr.smithi097.swvonl (mgr.14180) 390 : audit [DBG] from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:19.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:19 smithi097 bash[24958]: audit 2023-11-17T06:58:18.417670+0000 mon.smithi097 (mon.0) 727 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:19.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:19 smithi097 bash[24958]: cluster 2023-11-17T06:58:18.614980+0000 mgr.smithi097.swvonl (mgr.14180) 391 : cluster [DBG] pgmap v329: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:19 smithi193 bash[28559]: audit 2023-11-17T06:58:18.143873+0000 mgr.smithi097.swvonl (mgr.14180) 390 : audit [DBG] from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:19 smithi193 bash[28559]: audit 2023-11-17T06:58:18.417670+0000 mon.smithi097 (mon.0) 727 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:19.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:19 smithi193 bash[28559]: cluster 2023-11-17T06:58:18.614980+0000 mgr.smithi097.swvonl (mgr.14180) 391 : cluster [DBG] pgmap v329: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:20.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:22.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:21 smithi097 bash[24958]: cluster 2023-11-17T06:58:20.616183+0000 mgr.smithi097.swvonl (mgr.14180) 392 : cluster [DBG] pgmap v330: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:22.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:21 smithi193 bash[28559]: cluster 2023-11-17T06:58:20.616183+0000 mgr.smithi097.swvonl (mgr.14180) 392 : cluster [DBG] pgmap v330: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:23.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.363923+0000 mon.smithi097 (mon.0) 728 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: cluster 2023-11-17T06:58:22.616865+0000 mgr.smithi097.swvonl (mgr.14180) 393 : cluster [DBG] pgmap v331: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:23.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.808171+0000 mon.smithi097 (mon.0) 729 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.809408+0000 mon.smithi097 (mon.0) 730 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:58:23.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.810537+0000 mon.smithi097 (mon.0) 731 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:58:23.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.816082+0000 mon.smithi097 (mon.0) 732 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.821231+0000 mon.smithi097 (mon.0) 733 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:58:23.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:23 smithi097 bash[24958]: audit 2023-11-17T06:58:22.834878+0000 mon.smithi097 (mon.0) 734 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.714 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:23.714 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:01.577189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xnueuf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:12.408554Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\nDeploy daemon haproxy.nfs.foo.smithi193.rddnon ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.609373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vtwats on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\nDeploy daemon haproxy.nfs.foo.smithi097.vtwats ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.611669Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xfospj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.363923+0000 mon.smithi097 (mon.0) 728 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: cluster 2023-11-17T06:58:22.616865+0000 mgr.smithi097.swvonl (mgr.14180) 393 : cluster [DBG] pgmap v331: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.808171+0000 mon.smithi097 (mon.0) 729 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.809408+0000 mon.smithi097 (mon.0) 730 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.810537+0000 mon.smithi097 (mon.0) 731 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.816082+0000 mon.smithi097 (mon.0) 732 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.821231+0000 mon.smithi097 (mon.0) 733 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:58:23.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:23 smithi193 bash[28559]: audit 2023-11-17T06:58:22.834878+0000 mon.smithi097 (mon.0) 734 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:24.591 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:58:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:24 smithi097 bash[24958]: cluster 2023-11-17T06:58:22.816750+0000 mgr.smithi097.swvonl (mgr.14180) 394 : cluster [DBG] pgmap v332: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2023-11-17T06:58:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:24 smithi097 bash[24958]: cephadm 2023-11-17T06:58:22.837625+0000 mgr.smithi097.swvonl (mgr.14180) 395 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.howxaw on smithi193 2023-11-17T06:58:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:24 smithi097 bash[24958]: cluster 2023-11-17T06:58:23.361691+0000 mon.smithi097 (mon.0) 735 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:58:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:24 smithi097 bash[24958]: cluster 2023-11-17T06:58:23.361754+0000 mon.smithi097 (mon.0) 736 : cluster [INF] Cluster is now healthy 2023-11-17T06:58:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:24 smithi097 bash[24958]: audit 2023-11-17T06:58:23.699834+0000 mgr.smithi097.swvonl (mgr.14180) 396 : audit [DBG] from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:24 smithi193 bash[28559]: cluster 2023-11-17T06:58:22.816750+0000 mgr.smithi097.swvonl (mgr.14180) 394 : cluster [DBG] pgmap v332: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2023-11-17T06:58:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:24 smithi193 bash[28559]: cephadm 2023-11-17T06:58:22.837625+0000 mgr.smithi097.swvonl (mgr.14180) 395 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.howxaw on smithi193 2023-11-17T06:58:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:24 smithi193 bash[28559]: cluster 2023-11-17T06:58:23.361691+0000 mon.smithi097 (mon.0) 735 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:58:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:24 smithi193 bash[28559]: cluster 2023-11-17T06:58:23.361754+0000 mon.smithi097 (mon.0) 736 : cluster [INF] Cluster is now healthy 2023-11-17T06:58:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:24 smithi193 bash[28559]: audit 2023-11-17T06:58:23.699834+0000 mgr.smithi097.swvonl (mgr.14180) 396 : audit [DBG] from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:25.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:26.552 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:26 smithi193 bash[28559]: cluster 2023-11-17T06:58:24.817864+0000 mgr.smithi097.swvonl (mgr.14180) 397 : cluster [DBG] pgmap v333: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2023-11-17T06:58:26.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:26 smithi097 bash[24958]: cluster 2023-11-17T06:58:24.817864+0000 mgr.smithi097.swvonl (mgr.14180) 397 : cluster [DBG] pgmap v333: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 246 B/s rd, 0 op/s 2023-11-17T06:58:27.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: cephadm 2023-11-17T06:58:26.620995+0000 mgr.smithi097.swvonl (mgr.14180) 398 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:58:27.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: cephadm 2023-11-17T06:58:26.621468+0000 mgr.smithi097.swvonl (mgr.14180) 399 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: audit 2023-11-17T06:58:26.622123+0000 mon.smithi097 (mon.0) 737 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.howxaw"}]: dispatch 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: cephadm 2023-11-17T06:58:26.623683+0000 mgr.smithi097.swvonl (mgr.14180) 400 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:27 smithi097 bash[24958]: cephadm 2023-11-17T06:58:26.627865+0000 mgr.smithi097.swvonl (mgr.14180) 401 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.evrcay on smithi097 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: cephadm 2023-11-17T06:58:26.620995+0000 mgr.smithi097.swvonl (mgr.14180) 398 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: cephadm 2023-11-17T06:58:26.621468+0000 mgr.smithi097.swvonl (mgr.14180) 399 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: audit 2023-11-17T06:58:26.622123+0000 mon.smithi097 (mon.0) 737 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.howxaw"}]: dispatch 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: cephadm 2023-11-17T06:58:26.623683+0000 mgr.smithi097.swvonl (mgr.14180) 400 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.howxaw ... 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:27.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:27 smithi193 bash[28559]: cephadm 2023-11-17T06:58:26.627865+0000 mgr.smithi097.swvonl (mgr.14180) 401 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.evrcay on smithi097 2023-11-17T06:58:28.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:28 smithi097 bash[24958]: cluster 2023-11-17T06:58:26.818625+0000 mgr.smithi097.swvonl (mgr.14180) 402 : cluster [DBG] pgmap v334: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T06:58:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:28 smithi193 bash[28559]: cluster 2023-11-17T06:58:26.818625+0000 mgr.smithi097.swvonl (mgr.14180) 402 : cluster [DBG] pgmap v334: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T06:58:29.253 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:29.253 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:12.408554Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rddnon on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rddnon\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rddnon\nDeploy daemon haproxy.nfs.foo.smithi193.rddnon ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.609373Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vtwats on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vtwats\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vtwats\nDeploy daemon haproxy.nfs.foo.smithi097.vtwats ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:16.611669Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xfospj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:30.156 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:58:30.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:30 smithi097 bash[24958]: cluster 2023-11-17T06:58:28.819463+0000 mgr.smithi097.swvonl (mgr.14180) 403 : cluster [DBG] pgmap v335: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T06:58:30.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:30 smithi097 bash[24958]: audit 2023-11-17T06:58:29.236253+0000 mgr.smithi097.swvonl (mgr.14180) 404 : audit [DBG] from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:30.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:30 smithi097 bash[24958]: audit 2023-11-17T06:58:30.336655+0000 mon.smithi097 (mon.0) 738 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.evrcay"}]: dispatch 2023-11-17T06:58:30.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:30 smithi193 bash[28559]: cluster 2023-11-17T06:58:28.819463+0000 mgr.smithi097.swvonl (mgr.14180) 403 : cluster [DBG] pgmap v335: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T06:58:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:30 smithi193 bash[28559]: audit 2023-11-17T06:58:29.236253+0000 mgr.smithi097.swvonl (mgr.14180) 404 : audit [DBG] from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:30.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:30 smithi193 bash[28559]: audit 2023-11-17T06:58:30.336655+0000 mon.smithi097 (mon.0) 738 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.evrcay"}]: dispatch 2023-11-17T06:58:31.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:31.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cephadm 2023-11-17T06:58:30.335739+0000 mgr.smithi097.swvonl (mgr.14180) 405 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:31.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cephadm 2023-11-17T06:58:30.336165+0000 mgr.smithi097.swvonl (mgr.14180) 406 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cephadm 2023-11-17T06:58:30.337725+0000 mgr.smithi097.swvonl (mgr.14180) 407 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cephadm 2023-11-17T06:58:30.341791+0000 mgr.smithi097.swvonl (mgr.14180) 408 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cephadm 2023-11-17T06:58:30.344858+0000 mgr.smithi097.swvonl (mgr.14180) 409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cluster 2023-11-17T06:58:30.346200+0000 mgr.smithi097.swvonl (mgr.14180) 410 : cluster [DBG] pgmap v336: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2023-11-17T06:58:31.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:31 smithi097 bash[24958]: cluster 2023-11-17T06:58:30.391388+0000 mon.smithi097 (mon.0) 739 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:31.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cephadm 2023-11-17T06:58:30.335739+0000 mgr.smithi097.swvonl (mgr.14180) 405 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:58:31.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cephadm 2023-11-17T06:58:30.336165+0000 mgr.smithi097.swvonl (mgr.14180) 406 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cephadm 2023-11-17T06:58:30.337725+0000 mgr.smithi097.swvonl (mgr.14180) 407 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.evrcay ... 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:58:31.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:58:31.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cephadm 2023-11-17T06:58:30.341791+0000 mgr.smithi097.swvonl (mgr.14180) 408 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:31.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cephadm 2023-11-17T06:58:30.344858+0000 mgr.smithi097.swvonl (mgr.14180) 409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:58:31.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cluster 2023-11-17T06:58:30.346200+0000 mgr.smithi097.swvonl (mgr.14180) 410 : cluster [DBG] pgmap v336: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2023-11-17T06:58:31.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:31 smithi193 bash[28559]: cluster 2023-11-17T06:58:30.391388+0000 mon.smithi097 (mon.0) 739 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:58:33.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:33 smithi097 bash[24958]: cluster 2023-11-17T06:58:32.347028+0000 mgr.smithi097.swvonl (mgr.14180) 411 : cluster [DBG] pgmap v337: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2023-11-17T06:58:33.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:33 smithi193 bash[28559]: cluster 2023-11-17T06:58:32.347028+0000 mgr.smithi097.swvonl (mgr.14180) 411 : cluster [DBG] pgmap v337: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2023-11-17T06:58:34.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:34 smithi097 bash[24958]: audit 2023-11-17T06:58:33.435238+0000 mon.smithi097 (mon.0) 740 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:34.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:34 smithi193 bash[28559]: audit 2023-11-17T06:58:33.435238+0000 mon.smithi097 (mon.0) 740 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:34.839 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:34.839 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:35.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:35 smithi193 bash[28559]: cluster 2023-11-17T06:58:34.348075+0000 mgr.smithi097.swvonl (mgr.14180) 412 : cluster [DBG] pgmap v338: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:35.655 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:58:35.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:35 smithi097 bash[24958]: cluster 2023-11-17T06:58:34.348075+0000 mgr.smithi097.swvonl (mgr.14180) 412 : cluster [DBG] pgmap v338: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:36.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:36.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:36 smithi097 bash[24958]: audit 2023-11-17T06:58:34.829958+0000 mgr.smithi097.swvonl (mgr.14180) 413 : audit [DBG] from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:36.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:36 smithi193 bash[28559]: audit 2023-11-17T06:58:34.829958+0000 mgr.smithi097.swvonl (mgr.14180) 413 : audit [DBG] from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:37.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:37 smithi097 bash[24958]: cluster 2023-11-17T06:58:36.349527+0000 mgr.smithi097.swvonl (mgr.14180) 414 : cluster [DBG] pgmap v339: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:37.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:37 smithi193 bash[28559]: cluster 2023-11-17T06:58:36.349527+0000 mgr.smithi097.swvonl (mgr.14180) 414 : cluster [DBG] pgmap v339: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:39.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:39 smithi097 bash[24958]: cluster 2023-11-17T06:58:38.350602+0000 mgr.smithi097.swvonl (mgr.14180) 415 : cluster [DBG] pgmap v340: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:39.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:39 smithi193 bash[28559]: cluster 2023-11-17T06:58:38.350602+0000 mgr.smithi097.swvonl (mgr.14180) 415 : cluster [DBG] pgmap v340: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T06:58:40.276 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:58:40.276 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:58:41.152 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:58:41.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:41 smithi097 bash[24958]: audit 2023-11-17T06:58:40.266765+0000 mgr.smithi097.swvonl (mgr.14180) 416 : audit [DBG] from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:41.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:41 smithi097 bash[24958]: cluster 2023-11-17T06:58:40.351922+0000 mgr.smithi097.swvonl (mgr.14180) 417 : cluster [DBG] pgmap v341: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 355 B/s rd, 0 op/s 2023-11-17T06:58:41.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:41 smithi193 bash[28559]: audit 2023-11-17T06:58:40.266765+0000 mgr.smithi097.swvonl (mgr.14180) 416 : audit [DBG] from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:58:41.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:41 smithi193 bash[28559]: cluster 2023-11-17T06:58:40.351922+0000 mgr.smithi097.swvonl (mgr.14180) 417 : cluster [DBG] pgmap v341: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 355 B/s rd, 0 op/s 2023-11-17T06:58:42.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:58:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:42 smithi097 bash[24958]: audit 2023-11-17T06:58:41.515161+0000 mon.smithi097 (mon.0) 741 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:42.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:42 smithi193 bash[28559]: audit 2023-11-17T06:58:41.515161+0000 mon.smithi097 (mon.0) 741 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:58:43.628 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:43 smithi097 bash[24958]: cluster 2023-11-17T06:58:42.352626+0000 mgr.smithi097.swvonl (mgr.14180) 418 : cluster [DBG] pgmap v342: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:58:43.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:43 smithi193 bash[28559]: cluster 2023-11-17T06:58:42.352626+0000 mgr.smithi097.swvonl (mgr.14180) 418 : cluster [DBG] pgmap v342: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:58:45.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:45 smithi193 bash[28559]: cluster 2023-11-17T06:58:44.353736+0000 mgr.smithi097.swvonl (mgr.14180) 419 : cluster [DBG] pgmap v343: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.774 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:45 smithi097 bash[24958]: cluster 2023-11-17T06:58:44.353736+0000 mgr.smithi097.swvonl (mgr.14180) 419 : cluster [DBG] pgmap v343: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.778 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:04.778 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:04.812 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:47 smithi193 bash[28559]: audit 2023-11-17T06:58:45.897645+0000 mgr.smithi097.swvonl (mgr.14180) 420 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:47 smithi193 bash[28559]: cluster 2023-11-17T06:58:46.355093+0000 mgr.smithi097.swvonl (mgr.14180) 421 : cluster [DBG] pgmap v344: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:49 smithi193 bash[28559]: cluster 2023-11-17T06:58:48.356146+0000 mgr.smithi097.swvonl (mgr.14180) 422 : cluster [DBG] pgmap v345: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:51 smithi193 bash[28559]: cluster 2023-11-17T06:58:50.357513+0000 mgr.smithi097.swvonl (mgr.14180) 423 : cluster [DBG] pgmap v346: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:53 smithi193 bash[28559]: cluster 2023-11-17T06:58:52.358483+0000 mgr.smithi097.swvonl (mgr.14180) 424 : cluster [DBG] pgmap v347: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:55 smithi193 bash[28559]: cluster 2023-11-17T06:58:54.359595+0000 mgr.smithi097.swvonl (mgr.14180) 425 : cluster [DBG] pgmap v348: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:57 smithi193 bash[28559]: cluster 2023-11-17T06:58:56.360888+0000 mgr.smithi097.swvonl (mgr.14180) 426 : cluster [DBG] pgmap v349: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:58:59 smithi193 bash[28559]: cluster 2023-11-17T06:58:58.362009+0000 mgr.smithi097.swvonl (mgr.14180) 427 : cluster [DBG] pgmap v350: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:01 smithi193 bash[28559]: cluster 2023-11-17T06:59:00.363357+0000 mgr.smithi097.swvonl (mgr.14180) 428 : cluster [DBG] pgmap v351: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:04.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:03 smithi193 bash[28559]: cluster 2023-11-17T06:59:02.364327+0000 mgr.smithi097.swvonl (mgr.14180) 429 : cluster [DBG] pgmap v352: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:47 smithi097 bash[24958]: audit 2023-11-17T06:58:45.897645+0000 mgr.smithi097.swvonl (mgr.14180) 420 : audit [DBG] from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:47 smithi097 bash[24958]: cluster 2023-11-17T06:58:46.355093+0000 mgr.smithi097.swvonl (mgr.14180) 421 : cluster [DBG] pgmap v344: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:49 smithi097 bash[24958]: cluster 2023-11-17T06:58:48.356146+0000 mgr.smithi097.swvonl (mgr.14180) 422 : cluster [DBG] pgmap v345: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:51 smithi097 bash[24958]: cluster 2023-11-17T06:58:50.357513+0000 mgr.smithi097.swvonl (mgr.14180) 423 : cluster [DBG] pgmap v346: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:53 smithi097 bash[24958]: cluster 2023-11-17T06:58:52.358483+0000 mgr.smithi097.swvonl (mgr.14180) 424 : cluster [DBG] pgmap v347: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:55 smithi097 bash[24958]: cluster 2023-11-17T06:58:54.359595+0000 mgr.smithi097.swvonl (mgr.14180) 425 : cluster [DBG] pgmap v348: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.814 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:57 smithi097 bash[24958]: cluster 2023-11-17T06:58:56.360888+0000 mgr.smithi097.swvonl (mgr.14180) 426 : cluster [DBG] pgmap v349: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:58:59 smithi097 bash[24958]: cluster 2023-11-17T06:58:58.362009+0000 mgr.smithi097.swvonl (mgr.14180) 427 : cluster [DBG] pgmap v350: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:04.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:01 smithi097 bash[24958]: cluster 2023-11-17T06:59:00.363357+0000 mgr.smithi097.swvonl (mgr.14180) 428 : cluster [DBG] pgmap v351: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:04.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:03 smithi097 bash[24958]: cluster 2023-11-17T06:59:02.364327+0000 mgr.smithi097.swvonl (mgr.14180) 429 : cluster [DBG] pgmap v352: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:05.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:05 smithi193 bash[28559]: cluster 2023-11-17T06:59:04.365373+0000 mgr.smithi097.swvonl (mgr.14180) 430 : cluster [DBG] pgmap v353: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:05.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:05 smithi097 bash[24958]: cluster 2023-11-17T06:59:04.365373+0000 mgr.smithi097.swvonl (mgr.14180) 430 : cluster [DBG] pgmap v353: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:05.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:07 smithi097 bash[24958]: cluster 2023-11-17T06:59:06.366736+0000 mgr.smithi097.swvonl (mgr.14180) 431 : cluster [DBG] pgmap v354: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:07 smithi193 bash[28559]: cluster 2023-11-17T06:59:06.366736+0000 mgr.smithi097.swvonl (mgr.14180) 431 : cluster [DBG] pgmap v354: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:09.360 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:09.361 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:09.663 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:09 smithi097 bash[24958]: cluster 2023-11-17T06:59:08.367631+0000 mgr.smithi097.swvonl (mgr.14180) 432 : cluster [DBG] pgmap v355: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:09.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:09 smithi193 bash[28559]: cluster 2023-11-17T06:59:08.367631+0000 mgr.smithi097.swvonl (mgr.14180) 432 : cluster [DBG] pgmap v355: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:10.299 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:10.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:10 smithi097 bash[24958]: audit 2023-11-17T06:59:09.347582+0000 mgr.smithi097.swvonl (mgr.14180) 433 : audit [DBG] from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:10 smithi193 bash[28559]: audit 2023-11-17T06:59:09.347582+0000 mgr.smithi097.swvonl (mgr.14180) 433 : audit [DBG] from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:11.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:11.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:11 smithi097 bash[24958]: cluster 2023-11-17T06:59:10.369055+0000 mgr.smithi097.swvonl (mgr.14180) 434 : cluster [DBG] pgmap v356: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:11.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:11 smithi193 bash[28559]: cluster 2023-11-17T06:59:10.369055+0000 mgr.smithi097.swvonl (mgr.14180) 434 : cluster [DBG] pgmap v356: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:13.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:13 smithi097 bash[24958]: cluster 2023-11-17T06:59:12.370039+0000 mgr.smithi097.swvonl (mgr.14180) 435 : cluster [DBG] pgmap v357: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:13.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:13 smithi193 bash[28559]: cluster 2023-11-17T06:59:12.370039+0000 mgr.smithi097.swvonl (mgr.14180) 435 : cluster [DBG] pgmap v357: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:14.679 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:14.679 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:15.528 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:15.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:15 smithi193 bash[28559]: cluster 2023-11-17T06:59:14.371085+0000 mgr.smithi097.swvonl (mgr.14180) 436 : cluster [DBG] pgmap v358: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:15.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:15 smithi193 bash[28559]: audit 2023-11-17T06:59:14.663149+0000 mgr.smithi097.swvonl (mgr.14180) 437 : audit [DBG] from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:15.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:15 smithi097 bash[24958]: cluster 2023-11-17T06:59:14.371085+0000 mgr.smithi097.swvonl (mgr.14180) 436 : cluster [DBG] pgmap v358: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:15.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:15 smithi097 bash[24958]: audit 2023-11-17T06:59:14.663149+0000 mgr.smithi097.swvonl (mgr.14180) 437 : audit [DBG] from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:16.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:17.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:17 smithi097 bash[24958]: cluster 2023-11-17T06:59:16.372422+0000 mgr.smithi097.swvonl (mgr.14180) 438 : cluster [DBG] pgmap v359: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:17.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:17 smithi193 bash[28559]: cluster 2023-11-17T06:59:16.372422+0000 mgr.smithi097.swvonl (mgr.14180) 438 : cluster [DBG] pgmap v359: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:19.759 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:19 smithi097 bash[24958]: cluster 2023-11-17T06:59:18.373201+0000 mgr.smithi097.swvonl (mgr.14180) 439 : cluster [DBG] pgmap v360: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:19.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:19 smithi193 bash[28559]: cluster 2023-11-17T06:59:18.373201+0000 mgr.smithi097.swvonl (mgr.14180) 439 : cluster [DBG] pgmap v360: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:20.128 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:20.129 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:21.013 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:21.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:21 smithi097 bash[24958]: audit 2023-11-17T06:59:20.119322+0000 mgr.smithi097.swvonl (mgr.14180) 440 : audit [DBG] from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:21.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:21 smithi097 bash[24958]: cluster 2023-11-17T06:59:20.374635+0000 mgr.smithi097.swvonl (mgr.14180) 441 : cluster [DBG] pgmap v361: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:21.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:21 smithi193 bash[28559]: audit 2023-11-17T06:59:20.119322+0000 mgr.smithi097.swvonl (mgr.14180) 440 : audit [DBG] from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:21.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:21 smithi193 bash[28559]: cluster 2023-11-17T06:59:20.374635+0000 mgr.smithi097.swvonl (mgr.14180) 441 : cluster [DBG] pgmap v361: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:22.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:23.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:23 smithi097 bash[24958]: cluster 2023-11-17T06:59:22.375566+0000 mgr.smithi097.swvonl (mgr.14180) 442 : cluster [DBG] pgmap v362: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:23.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:23 smithi193 bash[28559]: cluster 2023-11-17T06:59:22.375566+0000 mgr.smithi097.swvonl (mgr.14180) 442 : cluster [DBG] pgmap v362: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:25.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:25 smithi193 bash[28559]: cluster 2023-11-17T06:59:24.376629+0000 mgr.smithi097.swvonl (mgr.14180) 443 : cluster [DBG] pgmap v363: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:25.575 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:30.155 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:30.156 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:25 smithi097 bash[24958]: cluster 2023-11-17T06:59:24.376629+0000 mgr.smithi097.swvonl (mgr.14180) 443 : cluster [DBG] pgmap v363: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:30.156 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:26 smithi193 bash[28559]: audit 2023-11-17T06:59:25.560724+0000 mgr.smithi097.swvonl (mgr.14180) 444 : audit [DBG] from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:30.157 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:27 smithi193 bash[28559]: cluster 2023-11-17T06:59:26.377925+0000 mgr.smithi097.swvonl (mgr.14180) 445 : cluster [DBG] pgmap v364: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:30.157 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:29 smithi193 bash[28559]: cluster 2023-11-17T06:59:28.378942+0000 mgr.smithi097.swvonl (mgr.14180) 446 : cluster [DBG] pgmap v365: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:30.157 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:30.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:26 smithi097 bash[24958]: audit 2023-11-17T06:59:25.560724+0000 mgr.smithi097.swvonl (mgr.14180) 444 : audit [DBG] from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:30.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:27 smithi097 bash[24958]: cluster 2023-11-17T06:59:26.377925+0000 mgr.smithi097.swvonl (mgr.14180) 445 : cluster [DBG] pgmap v364: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:30.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:29 smithi097 bash[24958]: cluster 2023-11-17T06:59:28.378942+0000 mgr.smithi097.swvonl (mgr.14180) 446 : cluster [DBG] pgmap v365: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:30.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:30 smithi097 bash[24958]: audit 2023-11-17T06:59:30.349703+0000 mon.smithi097 (mon.0) 742 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:59:30.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:30 smithi193 bash[28559]: audit 2023-11-17T06:59:30.349703+0000 mon.smithi097 (mon.0) 742 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T06:59:31.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:32.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: cluster 2023-11-17T06:59:30.379937+0000 mgr.smithi097.swvonl (mgr.14180) 447 : cluster [DBG] pgmap v366: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:32.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.629116+0000 mon.smithi097 (mon.0) 743 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:32.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.921673+0000 mon.smithi097 (mon.0) 744 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:59:32.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.923572+0000 mon.smithi097 (mon.0) 745 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:59:32.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.931505+0000 mon.smithi097 (mon.0) 746 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:32.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.939077+0000 mon.smithi097 (mon.0) 747 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:59:32.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:31 smithi097 bash[24958]: audit 2023-11-17T06:59:30.951178+0000 mon.smithi097 (mon.0) 748 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: cluster 2023-11-17T06:59:30.379937+0000 mgr.smithi097.swvonl (mgr.14180) 447 : cluster [DBG] pgmap v366: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:32.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.629116+0000 mon.smithi097 (mon.0) 743 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:32.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.921673+0000 mon.smithi097 (mon.0) 744 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T06:59:32.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.923572+0000 mon.smithi097 (mon.0) 745 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T06:59:32.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.931505+0000 mon.smithi097 (mon.0) 746 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:32.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.939077+0000 mon.smithi097 (mon.0) 747 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T06:59:32.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:31 smithi193 bash[28559]: audit 2023-11-17T06:59:30.951178+0000 mon.smithi097 (mon.0) 748 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:33.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:32 smithi097 bash[24958]: cluster 2023-11-17T06:59:30.932318+0000 mgr.smithi097.swvonl (mgr.14180) 448 : cluster [DBG] pgmap v367: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:33.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:32 smithi097 bash[24958]: cephadm 2023-11-17T06:59:30.954090+0000 mgr.smithi097.swvonl (mgr.14180) 449 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.wrkhtn on smithi193 2023-11-17T06:59:33.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:32 smithi097 bash[24958]: cluster 2023-11-17T06:59:31.929086+0000 mon.smithi097 (mon.0) 749 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:59:33.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:32 smithi097 bash[24958]: cluster 2023-11-17T06:59:31.929153+0000 mon.smithi097 (mon.0) 750 : cluster [INF] Cluster is now healthy 2023-11-17T06:59:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:32 smithi193 bash[28559]: cluster 2023-11-17T06:59:30.932318+0000 mgr.smithi097.swvonl (mgr.14180) 448 : cluster [DBG] pgmap v367: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:33.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:32 smithi193 bash[28559]: cephadm 2023-11-17T06:59:30.954090+0000 mgr.smithi097.swvonl (mgr.14180) 449 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.wrkhtn on smithi193 2023-11-17T06:59:33.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:32 smithi193 bash[28559]: cluster 2023-11-17T06:59:31.929086+0000 mon.smithi097 (mon.0) 749 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T06:59:33.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:32 smithi193 bash[28559]: cluster 2023-11-17T06:59:31.929153+0000 mon.smithi097 (mon.0) 750 : cluster [INF] Cluster is now healthy 2023-11-17T06:59:34.618 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:34.618 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:16.613195Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.erapuu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:26.623445Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.howxaw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-howxaw\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.howxaw\nDeploy daemon haproxy.nfs.foo.smithi193.howxaw ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.337528Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.evrcay on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-evrcay\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.evrcay\nDeploy daemon haproxy.nfs.foo.smithi097.evrcay ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:58:30.341581Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.yfbdip on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:34.799 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:34 smithi193 bash[28559]: cluster 2023-11-17T06:59:32.932752+0000 mgr.smithi097.swvonl (mgr.14180) 450 : cluster [DBG] pgmap v368: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:34.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:34 smithi097 bash[24958]: cluster 2023-11-17T06:59:32.932752+0000 mgr.smithi097.swvonl (mgr.14180) 450 : cluster [DBG] pgmap v368: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:35.471 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:35.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:35 smithi097 bash[24958]: audit 2023-11-17T06:59:34.609518+0000 mgr.smithi097.swvonl (mgr.14180) 451 : audit [DBG] from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:35.815 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:35 smithi097 bash[24958]: audit 2023-11-17T06:59:34.877643+0000 mon.smithi097 (mon.0) 751 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.wrkhtn"}]: dispatch 2023-11-17T06:59:36.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:35 smithi193 bash[28559]: audit 2023-11-17T06:59:34.609518+0000 mgr.smithi097.swvonl (mgr.14180) 451 : audit [DBG] from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:36.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:35 smithi193 bash[28559]: audit 2023-11-17T06:59:34.877643+0000 mon.smithi097 (mon.0) 751 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.wrkhtn"}]: dispatch 2023-11-17T06:59:36.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:36.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: cephadm 2023-11-17T06:59:34.876475+0000 mgr.smithi097.swvonl (mgr.14180) 452 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: cephadm 2023-11-17T06:59:34.876980+0000 mgr.smithi097.swvonl (mgr.14180) 453 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: cephadm 2023-11-17T06:59:34.879222+0000 mgr.smithi097.swvonl (mgr.14180) 454 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:36.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: cephadm 2023-11-17T06:59:34.884198+0000 mgr.smithi097.swvonl (mgr.14180) 455 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.gsgxzb on smithi097 2023-11-17T06:59:36.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:36 smithi097 bash[24958]: cluster 2023-11-17T06:59:34.933665+0000 mgr.smithi097.swvonl (mgr.14180) 456 : cluster [DBG] pgmap v369: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: cephadm 2023-11-17T06:59:34.876475+0000 mgr.smithi097.swvonl (mgr.14180) 452 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: cephadm 2023-11-17T06:59:34.876980+0000 mgr.smithi097.swvonl (mgr.14180) 453 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: cephadm 2023-11-17T06:59:34.879222+0000 mgr.smithi097.swvonl (mgr.14180) 454 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: cephadm 2023-11-17T06:59:34.884198+0000 mgr.smithi097.swvonl (mgr.14180) 455 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.gsgxzb on smithi097 2023-11-17T06:59:37.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:36 smithi193 bash[28559]: cluster 2023-11-17T06:59:34.933665+0000 mgr.smithi097.swvonl (mgr.14180) 456 : cluster [DBG] pgmap v369: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:39.026 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:38 smithi097 bash[24958]: cluster 2023-11-17T06:59:36.934513+0000 mgr.smithi097.swvonl (mgr.14180) 457 : cluster [DBG] pgmap v370: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:39.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:38 smithi193 bash[28559]: cluster 2023-11-17T06:59:36.934513+0000 mgr.smithi097.swvonl (mgr.14180) 457 : cluster [DBG] pgmap v370: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T06:59:39.971 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:39 smithi097 bash[24958]: audit 2023-11-17T06:59:39.050417+0000 mon.smithi097 (mon.0) 752 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.gsgxzb"}]: dispatch 2023-11-17T06:59:40.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:39 smithi193 bash[28559]: audit 2023-11-17T06:59:39.050417+0000 mon.smithi097 (mon.0) 752 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.gsgxzb"}]: dispatch 2023-11-17T06:59:40.343 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:40.343 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:41.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cluster 2023-11-17T06:59:38.935417+0000 mgr.smithi097.swvonl (mgr.14180) 458 : cluster [DBG] pgmap v371: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T06:59:41.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cephadm 2023-11-17T06:59:39.049238+0000 mgr.smithi097.swvonl (mgr.14180) 459 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.057 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cephadm 2023-11-17T06:59:39.049855+0000 mgr.smithi097.swvonl (mgr.14180) 460 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cephadm 2023-11-17T06:59:39.051465+0000 mgr.smithi097.swvonl (mgr.14180) 461 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.058 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.059 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.059 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.059 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.060 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.060 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.060 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cephadm 2023-11-17T06:59:39.053282+0000 mgr.smithi097.swvonl (mgr.14180) 462 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:59:41.060 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cephadm 2023-11-17T06:59:39.054956+0000 mgr.smithi097.swvonl (mgr.14180) 463 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:59:41.060 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cluster 2023-11-17T06:59:39.055975+0000 mgr.smithi097.swvonl (mgr.14180) 464 : cluster [DBG] pgmap v372: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:59:41.061 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:40 smithi097 bash[24958]: cluster 2023-11-17T06:59:39.655700+0000 mon.smithi097 (mon.0) 753 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:59:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cluster 2023-11-17T06:59:38.935417+0000 mgr.smithi097.swvonl (mgr.14180) 458 : cluster [DBG] pgmap v371: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cephadm 2023-11-17T06:59:39.049238+0000 mgr.smithi097.swvonl (mgr.14180) 459 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cephadm 2023-11-17T06:59:39.049855+0000 mgr.smithi097.swvonl (mgr.14180) 460 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cephadm 2023-11-17T06:59:39.051465+0000 mgr.smithi097.swvonl (mgr.14180) 461 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T06:59:41.065 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cephadm 2023-11-17T06:59:39.053282+0000 mgr.smithi097.swvonl (mgr.14180) 462 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cephadm 2023-11-17T06:59:39.054956+0000 mgr.smithi097.swvonl (mgr.14180) 463 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cluster 2023-11-17T06:59:39.055975+0000 mgr.smithi097.swvonl (mgr.14180) 464 : cluster [DBG] pgmap v372: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T06:59:41.066 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:40 smithi193 bash[28559]: cluster 2023-11-17T06:59:39.655700+0000 mon.smithi097 (mon.0) 753 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T06:59:41.211 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:42.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:41 smithi097 bash[24958]: audit 2023-11-17T06:59:40.327522+0000 mgr.smithi097.swvonl (mgr.14180) 465 : audit [DBG] from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:42.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:41 smithi193 bash[28559]: audit 2023-11-17T06:59:40.327522+0000 mgr.smithi097.swvonl (mgr.14180) 465 : audit [DBG] from='client.14628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:42.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:43.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:42 smithi097 bash[24958]: cluster 2023-11-17T06:59:41.057307+0000 mgr.smithi097.swvonl (mgr.14180) 466 : cluster [DBG] pgmap v373: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:43.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:42 smithi193 bash[28559]: cluster 2023-11-17T06:59:41.057307+0000 mgr.smithi097.swvonl (mgr.14180) 466 : cluster [DBG] pgmap v373: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:44.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:44 smithi097 bash[24958]: cluster 2023-11-17T06:59:43.058331+0000 mgr.smithi097.swvonl (mgr.14180) 467 : cluster [DBG] pgmap v374: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:44.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:44 smithi097 bash[24958]: audit 2023-11-17T06:59:43.489966+0000 mon.smithi097 (mon.0) 754 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:44.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:44 smithi193 bash[28559]: cluster 2023-11-17T06:59:43.058331+0000 mgr.smithi097.swvonl (mgr.14180) 467 : cluster [DBG] pgmap v374: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:44.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:44 smithi193 bash[28559]: audit 2023-11-17T06:59:43.489966+0000 mon.smithi097 (mon.0) 754 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T06:59:45.874 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:45.875 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:46.694 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:46.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:46 smithi097 bash[24958]: cluster 2023-11-17T06:59:45.059248+0000 mgr.smithi097.swvonl (mgr.14180) 468 : cluster [DBG] pgmap v375: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:46.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:46 smithi193 bash[28559]: cluster 2023-11-17T06:59:45.059248+0000 mgr.smithi097.swvonl (mgr.14180) 468 : cluster [DBG] pgmap v375: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:47.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:47.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:47 smithi097 bash[24958]: audit 2023-11-17T06:59:45.860700+0000 mgr.smithi097.swvonl (mgr.14180) 469 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:47.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:47 smithi193 bash[28559]: audit 2023-11-17T06:59:45.860700+0000 mgr.smithi097.swvonl (mgr.14180) 469 : audit [DBG] from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:48.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:48 smithi097 bash[24958]: cluster 2023-11-17T06:59:47.060281+0000 mgr.smithi097.swvonl (mgr.14180) 470 : cluster [DBG] pgmap v376: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:48.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:48 smithi193 bash[28559]: cluster 2023-11-17T06:59:47.060281+0000 mgr.smithi097.swvonl (mgr.14180) 470 : cluster [DBG] pgmap v376: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:50.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:50 smithi097 bash[24958]: cluster 2023-11-17T06:59:49.061077+0000 mgr.smithi097.swvonl (mgr.14180) 471 : cluster [DBG] pgmap v377: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:50.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:50 smithi193 bash[28559]: cluster 2023-11-17T06:59:49.061077+0000 mgr.smithi097.swvonl (mgr.14180) 471 : cluster [DBG] pgmap v377: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T06:59:51.082 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:51.083 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:51.923 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:52.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:52 smithi097 bash[24958]: cluster 2023-11-17T06:59:51.061791+0000 mgr.smithi097.swvonl (mgr.14180) 472 : cluster [DBG] pgmap v378: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:52.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:52 smithi097 bash[24958]: audit 2023-11-17T06:59:51.068783+0000 mgr.smithi097.swvonl (mgr.14180) 473 : audit [DBG] from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:52.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:52 smithi193 bash[28559]: cluster 2023-11-17T06:59:51.061791+0000 mgr.smithi097.swvonl (mgr.14180) 472 : cluster [DBG] pgmap v378: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T06:59:52.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:52 smithi193 bash[28559]: audit 2023-11-17T06:59:51.068783+0000 mgr.smithi097.swvonl (mgr.14180) 473 : audit [DBG] from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:52.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:54.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:54 smithi097 bash[24958]: cluster 2023-11-17T06:59:53.062763+0000 mgr.smithi097.swvonl (mgr.14180) 474 : cluster [DBG] pgmap v379: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:56.836 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T06:59:56.836 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T06:59:56.836 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:54 smithi193 bash[28559]: cluster 2023-11-17T06:59:53.062763+0000 mgr.smithi097.swvonl (mgr.14180) 474 : cluster [DBG] pgmap v379: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:56.837 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:56 smithi193 bash[28559]: cluster 2023-11-17T06:59:55.064247+0000 mgr.smithi097.swvonl (mgr.14180) 475 : cluster [DBG] pgmap v380: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:56.837 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:56 smithi097 bash[24958]: cluster 2023-11-17T06:59:55.064247+0000 mgr.smithi097.swvonl (mgr.14180) 475 : cluster [DBG] pgmap v380: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:57.530 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T06:59:57.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:57 smithi097 bash[24958]: audit 2023-11-17T06:59:56.694910+0000 mgr.smithi097.swvonl (mgr.14180) 476 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:57.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:57 smithi193 bash[28559]: audit 2023-11-17T06:59:56.694910+0000 mgr.smithi097.swvonl (mgr.14180) 476 : audit [DBG] from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T06:59:58.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T06:59:58.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 06:59:58 smithi097 bash[24958]: cluster 2023-11-17T06:59:57.065352+0000 mgr.smithi097.swvonl (mgr.14180) 477 : cluster [DBG] pgmap v381: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T06:59:58.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 06:59:58 smithi193 bash[28559]: cluster 2023-11-17T06:59:57.065352+0000 mgr.smithi097.swvonl (mgr.14180) 477 : cluster [DBG] pgmap v381: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:00.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T06:59:59.066467+0000 mgr.smithi097.swvonl (mgr.14180) 478 : cluster [DBG] pgmap v382: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000207+0000 mon.smithi097 (mon.0) 755 : cluster [WRN] Health detail: HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000259+0000 mon.smithi097 (mon.0) 756 : cluster [WRN] [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 4 daemon(s) 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000279+0000 mon.smithi097 (mon.0) 757 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000303+0000 mon.smithi097 (mon.0) 758 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000330+0000 mon.smithi097 (mon.0) 759 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000379+0000 mon.smithi097 (mon.0) 760 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000409+0000 mon.smithi097 (mon.0) 761 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000437+0000 mon.smithi097 (mon.0) 762 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000461+0000 mon.smithi097 (mon.0) 763 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000489+0000 mon.smithi097 (mon.0) 764 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000511+0000 mon.smithi097 (mon.0) 765 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000535+0000 mon.smithi097 (mon.0) 766 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000566+0000 mon.smithi097 (mon.0) 767 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000589+0000 mon.smithi097 (mon.0) 768 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000609+0000 mon.smithi097 (mon.0) 769 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000636+0000 mon.smithi097 (mon.0) 770 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000660+0000 mon.smithi097 (mon.0) 771 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000684+0000 mon.smithi097 (mon.0) 772 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000703+0000 mon.smithi097 (mon.0) 773 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000722+0000 mon.smithi097 (mon.0) 774 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000744+0000 mon.smithi097 (mon.0) 775 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000766+0000 mon.smithi097 (mon.0) 776 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000792+0000 mon.smithi097 (mon.0) 777 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:00.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000815+0000 mon.smithi097 (mon.0) 778 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:00.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000843+0000 mon.smithi097 (mon.0) 779 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T07:00:00.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000870+0000 mon.smithi097 (mon.0) 780 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000925+0000 mon.smithi097 (mon.0) 781 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:00.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:00.000941+0000 mon.smithi097 (mon.0) 782 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T06:59:59.066467+0000 mgr.smithi097.swvonl (mgr.14180) 478 : cluster [DBG] pgmap v382: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000207+0000 mon.smithi097 (mon.0) 755 : cluster [WRN] Health detail: HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000259+0000 mon.smithi097 (mon.0) 756 : cluster [WRN] [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 4 daemon(s) 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000279+0000 mon.smithi097 (mon.0) 757 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000303+0000 mon.smithi097 (mon.0) 758 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000330+0000 mon.smithi097 (mon.0) 759 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000379+0000 mon.smithi097 (mon.0) 760 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000409+0000 mon.smithi097 (mon.0) 761 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000437+0000 mon.smithi097 (mon.0) 762 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000461+0000 mon.smithi097 (mon.0) 763 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi193.wrkhtn ... 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000489+0000 mon.smithi097 (mon.0) 764 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000511+0000 mon.smithi097 (mon.0) 765 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000535+0000 mon.smithi097 (mon.0) 766 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000566+0000 mon.smithi097 (mon.0) 767 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000589+0000 mon.smithi097 (mon.0) 768 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000609+0000 mon.smithi097 (mon.0) 769 : cluster [WRN] Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000636+0000 mon.smithi097 (mon.0) 770 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000660+0000 mon.smithi097 (mon.0) 771 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000684+0000 mon.smithi097 (mon.0) 772 : cluster [WRN] Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000703+0000 mon.smithi097 (mon.0) 773 : cluster [WRN] /usr/bin/docker: stdout 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000722+0000 mon.smithi097 (mon.0) 774 : cluster [WRN] /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000744+0000 mon.smithi097 (mon.0) 775 : cluster [WRN] Deploy daemon haproxy.nfs.foo.smithi097.gsgxzb ... 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000766+0000 mon.smithi097 (mon.0) 776 : cluster [WRN] Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000792+0000 mon.smithi097 (mon.0) 777 : cluster [WRN] stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000815+0000 mon.smithi097 (mon.0) 778 : cluster [WRN] stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000843+0000 mon.smithi097 (mon.0) 779 : cluster [WRN] stat: stderr See 'docker run --help'. 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000870+0000 mon.smithi097 (mon.0) 780 : cluster [WRN] ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:00.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000925+0000 mon.smithi097 (mon.0) 781 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:00.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:00.000941+0000 mon.smithi097 (mon.0) 782 : cluster [WRN] Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:02.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:02.184 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:02.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:02 smithi097 bash[24958]: cluster 2023-11-17T07:00:01.067200+0000 mgr.smithi097.swvonl (mgr.14180) 479 : cluster [DBG] pgmap v383: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:03.048 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:03.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:02 smithi193 bash[28559]: cluster 2023-11-17T07:00:01.067200+0000 mgr.smithi097.swvonl (mgr.14180) 479 : cluster [DBG] pgmap v383: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:04.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:04.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:03 smithi097 bash[24958]: audit 2023-11-17T07:00:02.170380+0000 mgr.smithi097.swvonl (mgr.14180) 480 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:04.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:03 smithi193 bash[28559]: audit 2023-11-17T07:00:02.170380+0000 mgr.smithi097.swvonl (mgr.14180) 480 : audit [DBG] from='client.14644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:05.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:04 smithi097 bash[24958]: cluster 2023-11-17T07:00:03.068285+0000 mgr.smithi097.swvonl (mgr.14180) 481 : cluster [DBG] pgmap v384: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:05.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:04 smithi193 bash[28559]: cluster 2023-11-17T07:00:03.068285+0000 mgr.smithi097.swvonl (mgr.14180) 481 : cluster [DBG] pgmap v384: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:06.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:06 smithi097 bash[24958]: cluster 2023-11-17T07:00:05.069180+0000 mgr.smithi097.swvonl (mgr.14180) 482 : cluster [DBG] pgmap v385: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:07.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:06 smithi193 bash[28559]: cluster 2023-11-17T07:00:05.069180+0000 mgr.smithi097.swvonl (mgr.14180) 482 : cluster [DBG] pgmap v385: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:07.699 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:07.699 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:08.578 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:09.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:08 smithi097 bash[24958]: cluster 2023-11-17T07:00:07.070222+0000 mgr.smithi097.swvonl (mgr.14180) 483 : cluster [DBG] pgmap v386: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:09.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:08 smithi097 bash[24958]: audit 2023-11-17T07:00:07.684834+0000 mgr.smithi097.swvonl (mgr.14180) 484 : audit [DBG] from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:09.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:08 smithi193 bash[28559]: cluster 2023-11-17T07:00:07.070222+0000 mgr.smithi097.swvonl (mgr.14180) 483 : cluster [DBG] pgmap v386: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:09.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:08 smithi193 bash[28559]: audit 2023-11-17T07:00:07.684834+0000 mgr.smithi097.swvonl (mgr.14180) 484 : audit [DBG] from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:09.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:10.877 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:10 smithi097 bash[24958]: cluster 2023-11-17T07:00:09.071350+0000 mgr.smithi097.swvonl (mgr.14180) 485 : cluster [DBG] pgmap v387: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:11.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:10 smithi193 bash[28559]: cluster 2023-11-17T07:00:09.071350+0000 mgr.smithi097.swvonl (mgr.14180) 485 : cluster [DBG] pgmap v387: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:13.029 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:13.029 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:13.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:12 smithi097 bash[24958]: cluster 2023-11-17T07:00:11.072816+0000 mgr.smithi097.swvonl (mgr.14180) 486 : cluster [DBG] pgmap v388: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:13.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:12 smithi193 bash[28559]: cluster 2023-11-17T07:00:11.072816+0000 mgr.smithi097.swvonl (mgr.14180) 486 : cluster [DBG] pgmap v388: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:13.888 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:14.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:14.902 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:14 smithi097 bash[24958]: audit 2023-11-17T07:00:13.016131+0000 mgr.smithi097.swvonl (mgr.14180) 487 : audit [DBG] from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:14.902 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:14 smithi097 bash[24958]: cluster 2023-11-17T07:00:13.073869+0000 mgr.smithi097.swvonl (mgr.14180) 488 : cluster [DBG] pgmap v389: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:14 smithi193 bash[28559]: audit 2023-11-17T07:00:13.016131+0000 mgr.smithi097.swvonl (mgr.14180) 487 : audit [DBG] from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:14 smithi193 bash[28559]: cluster 2023-11-17T07:00:13.073869+0000 mgr.smithi097.swvonl (mgr.14180) 488 : cluster [DBG] pgmap v389: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:16.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:16 smithi097 bash[24958]: cluster 2023-11-17T07:00:15.075243+0000 mgr.smithi097.swvonl (mgr.14180) 489 : cluster [DBG] pgmap v390: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:16 smithi193 bash[28559]: cluster 2023-11-17T07:00:15.075243+0000 mgr.smithi097.swvonl (mgr.14180) 489 : cluster [DBG] pgmap v390: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:18.452 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:18.452 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:18.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:18 smithi097 bash[24958]: cluster 2023-11-17T07:00:17.076014+0000 mgr.smithi097.swvonl (mgr.14180) 490 : cluster [DBG] pgmap v391: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:18 smithi193 bash[28559]: cluster 2023-11-17T07:00:17.076014+0000 mgr.smithi097.swvonl (mgr.14180) 490 : cluster [DBG] pgmap v391: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:19.243 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:19 smithi097 bash[24958]: audit 2023-11-17T07:00:18.438060+0000 mgr.smithi097.swvonl (mgr.14180) 491 : audit [DBG] from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:20.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:19 smithi193 bash[28559]: audit 2023-11-17T07:00:18.438060+0000 mgr.smithi097.swvonl (mgr.14180) 491 : audit [DBG] from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:20.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:20.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:20 smithi097 bash[24958]: cluster 2023-11-17T07:00:19.077108+0000 mgr.smithi097.swvonl (mgr.14180) 492 : cluster [DBG] pgmap v392: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:20 smithi193 bash[28559]: cluster 2023-11-17T07:00:19.077108+0000 mgr.smithi097.swvonl (mgr.14180) 492 : cluster [DBG] pgmap v392: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:22.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:22 smithi097 bash[24958]: cluster 2023-11-17T07:00:21.077947+0000 mgr.smithi097.swvonl (mgr.14180) 493 : cluster [DBG] pgmap v393: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:23.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:22 smithi193 bash[28559]: cluster 2023-11-17T07:00:21.077947+0000 mgr.smithi097.swvonl (mgr.14180) 493 : cluster [DBG] pgmap v393: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:23.770 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:23.770 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:24.664 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:24 smithi097 bash[24958]: cluster 2023-11-17T07:00:23.078712+0000 mgr.smithi097.swvonl (mgr.14180) 494 : cluster [DBG] pgmap v394: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:24 smithi097 bash[24958]: audit 2023-11-17T07:00:23.761198+0000 mgr.smithi097.swvonl (mgr.14180) 495 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:24 smithi193 bash[28559]: cluster 2023-11-17T07:00:23.078712+0000 mgr.smithi097.swvonl (mgr.14180) 494 : cluster [DBG] pgmap v394: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:24 smithi193 bash[28559]: audit 2023-11-17T07:00:23.761198+0000 mgr.smithi097.swvonl (mgr.14180) 495 : audit [DBG] from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:25.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:27.037 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:26 smithi097 bash[24958]: cluster 2023-11-17T07:00:25.080099+0000 mgr.smithi097.swvonl (mgr.14180) 496 : cluster [DBG] pgmap v395: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:27.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:26 smithi193 bash[28559]: cluster 2023-11-17T07:00:25.080099+0000 mgr.smithi097.swvonl (mgr.14180) 496 : cluster [DBG] pgmap v395: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:29.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:28 smithi097 bash[24958]: cluster 2023-11-17T07:00:27.081225+0000 mgr.smithi097.swvonl (mgr.14180) 497 : cluster [DBG] pgmap v396: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:28 smithi193 bash[28559]: cluster 2023-11-17T07:00:27.081225+0000 mgr.smithi097.swvonl (mgr.14180) 497 : cluster [DBG] pgmap v396: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:29.170 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:29.170 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:29.999 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:31.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:31.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:30 smithi097 bash[24958]: cluster 2023-11-17T07:00:29.081965+0000 mgr.smithi097.swvonl (mgr.14180) 498 : cluster [DBG] pgmap v397: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:31.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:30 smithi097 bash[24958]: audit 2023-11-17T07:00:29.156191+0000 mgr.smithi097.swvonl (mgr.14180) 499 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:30 smithi193 bash[28559]: cluster 2023-11-17T07:00:29.081965+0000 mgr.smithi097.swvonl (mgr.14180) 498 : cluster [DBG] pgmap v397: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:30 smithi193 bash[28559]: audit 2023-11-17T07:00:29.156191+0000 mgr.smithi097.swvonl (mgr.14180) 499 : audit [DBG] from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:33.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:32 smithi097 bash[24958]: cluster 2023-11-17T07:00:31.083211+0000 mgr.smithi097.swvonl (mgr.14180) 500 : cluster [DBG] pgmap v398: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:32 smithi193 bash[28559]: cluster 2023-11-17T07:00:31.083211+0000 mgr.smithi097.swvonl (mgr.14180) 500 : cluster [DBG] pgmap v398: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:00:34.522 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:34.522 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:34 smithi097 bash[24958]: cluster 2023-11-17T07:00:33.084231+0000 mgr.smithi097.swvonl (mgr.14180) 501 : cluster [DBG] pgmap v399: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:35.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:34 smithi193 bash[28559]: cluster 2023-11-17T07:00:33.084231+0000 mgr.smithi097.swvonl (mgr.14180) 501 : cluster [DBG] pgmap v399: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:35.394 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:36.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:35 smithi097 bash[24958]: audit 2023-11-17T07:00:34.512900+0000 mgr.smithi097.swvonl (mgr.14180) 502 : audit [DBG] from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:36.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:35 smithi193 bash[28559]: audit 2023-11-17T07:00:34.512900+0000 mgr.smithi097.swvonl (mgr.14180) 502 : audit [DBG] from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:36.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:36 smithi097 bash[24958]: cluster 2023-11-17T07:00:35.085577+0000 mgr.smithi097.swvonl (mgr.14180) 503 : cluster [DBG] pgmap v400: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:36 smithi193 bash[28559]: cluster 2023-11-17T07:00:35.085577+0000 mgr.smithi097.swvonl (mgr.14180) 503 : cluster [DBG] pgmap v400: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:39.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:38 smithi097 bash[24958]: cluster 2023-11-17T07:00:37.086724+0000 mgr.smithi097.swvonl (mgr.14180) 504 : cluster [DBG] pgmap v401: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:39.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:38 smithi193 bash[28559]: cluster 2023-11-17T07:00:37.086724+0000 mgr.smithi097.swvonl (mgr.14180) 504 : cluster [DBG] pgmap v401: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:40.039 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:40.040 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:58:30.344684Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ssgsti on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.059009+0000 mon.smithi097 (mon.0) 783 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.362037+0000 mon.smithi097 (mon.0) 784 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.363643+0000 mon.smithi097 (mon.0) 785 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.371633+0000 mon.smithi097 (mon.0) 786 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.377556+0000 mon.smithi097 (mon.0) 787 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:00:40.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:39 smithi097 bash[24958]: audit 2023-11-17T07:00:39.386627+0000 mon.smithi097 (mon.0) 788 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:40.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.059009+0000 mon.smithi097 (mon.0) 783 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:00:40.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.362037+0000 mon.smithi097 (mon.0) 784 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:00:40.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.363643+0000 mon.smithi097 (mon.0) 785 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:00:40.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.371633+0000 mon.smithi097 (mon.0) 786 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:40.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.377556+0000 mon.smithi097 (mon.0) 787 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:00:40.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:39 smithi193 bash[28559]: audit 2023-11-17T07:00:39.386627+0000 mon.smithi097 (mon.0) 788 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:40.895 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:40 smithi097 bash[24958]: cluster 2023-11-17T07:00:39.087395+0000 mgr.smithi097.swvonl (mgr.14180) 505 : cluster [DBG] pgmap v402: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:40 smithi097 bash[24958]: cluster 2023-11-17T07:00:39.372260+0000 mgr.smithi097.swvonl (mgr.14180) 506 : cluster [DBG] pgmap v403: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:40 smithi097 bash[24958]: cephadm 2023-11-17T07:00:39.389285+0000 mgr.smithi097.swvonl (mgr.14180) 507 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.xphbfi on smithi193 2023-11-17T07:00:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:40 smithi097 bash[24958]: cluster 2023-11-17T07:00:39.675196+0000 mon.smithi097 (mon.0) 789 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:00:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:40 smithi097 bash[24958]: cluster 2023-11-17T07:00:39.675253+0000 mon.smithi097 (mon.0) 790 : cluster [INF] Cluster is now healthy 2023-11-17T07:00:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:40 smithi193 bash[28559]: cluster 2023-11-17T07:00:39.087395+0000 mgr.smithi097.swvonl (mgr.14180) 505 : cluster [DBG] pgmap v402: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:00:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:40 smithi193 bash[28559]: cluster 2023-11-17T07:00:39.372260+0000 mgr.smithi097.swvonl (mgr.14180) 506 : cluster [DBG] pgmap v403: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:40 smithi193 bash[28559]: cephadm 2023-11-17T07:00:39.389285+0000 mgr.smithi097.swvonl (mgr.14180) 507 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.xphbfi on smithi193 2023-11-17T07:00:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:40 smithi193 bash[28559]: cluster 2023-11-17T07:00:39.675196+0000 mon.smithi097 (mon.0) 789 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:00:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:40 smithi193 bash[28559]: cluster 2023-11-17T07:00:39.675253+0000 mon.smithi097 (mon.0) 790 : cluster [INF] Cluster is now healthy 2023-11-17T07:00:41.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:42.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:41 smithi097 bash[24958]: audit 2023-11-17T07:00:40.025143+0000 mgr.smithi097.swvonl (mgr.14180) 508 : audit [DBG] from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:42.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:41 smithi097 bash[24958]: audit 2023-11-17T07:00:41.518597+0000 mon.smithi097 (mon.0) 791 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:42.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:41 smithi193 bash[28559]: audit 2023-11-17T07:00:40.025143+0000 mgr.smithi097.swvonl (mgr.14180) 508 : audit [DBG] from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:42.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:41 smithi193 bash[28559]: audit 2023-11-17T07:00:41.518597+0000 mon.smithi097 (mon.0) 791 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:43.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:42 smithi097 bash[24958]: cluster 2023-11-17T07:00:41.373318+0000 mgr.smithi097.swvonl (mgr.14180) 509 : cluster [DBG] pgmap v404: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:43.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:42 smithi193 bash[28559]: cluster 2023-11-17T07:00:41.373318+0000 mgr.smithi097.swvonl (mgr.14180) 509 : cluster [DBG] pgmap v404: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:43.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:43 smithi097 bash[24958]: audit 2023-11-17T07:00:43.157146+0000 mon.smithi097 (mon.0) 792 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.xphbfi"}]: dispatch 2023-11-17T07:00:44.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:43 smithi193 bash[28559]: audit 2023-11-17T07:00:43.157146+0000 mon.smithi097 (mon.0) 792 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.xphbfi"}]: dispatch 2023-11-17T07:00:45.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: cephadm 2023-11-17T07:00:43.156070+0000 mgr.smithi097.swvonl (mgr.14180) 510 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:00:45.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: cephadm 2023-11-17T07:00:43.156411+0000 mgr.smithi097.swvonl (mgr.14180) 511 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: cephadm 2023-11-17T07:00:43.159083+0000 mgr.smithi097.swvonl (mgr.14180) 512 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: cephadm 2023-11-17T07:00:43.161984+0000 mgr.smithi097.swvonl (mgr.14180) 513 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ptseaz on smithi097 2023-11-17T07:00:45.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:44 smithi097 bash[24958]: cluster 2023-11-17T07:00:43.373943+0000 mgr.smithi097.swvonl (mgr.14180) 514 : cluster [DBG] pgmap v405: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:45.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: cephadm 2023-11-17T07:00:43.156070+0000 mgr.smithi097.swvonl (mgr.14180) 510 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: cephadm 2023-11-17T07:00:43.156411+0000 mgr.smithi097.swvonl (mgr.14180) 511 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: cephadm 2023-11-17T07:00:43.159083+0000 mgr.smithi097.swvonl (mgr.14180) 512 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi 2023-11-17T07:00:45.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.xphbfi ... 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: cephadm 2023-11-17T07:00:43.161984+0000 mgr.smithi097.swvonl (mgr.14180) 513 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ptseaz on smithi097 2023-11-17T07:00:45.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:44 smithi193 bash[28559]: cluster 2023-11-17T07:00:43.373943+0000 mgr.smithi097.swvonl (mgr.14180) 514 : cluster [DBG] pgmap v405: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:45.572 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:34.879009Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.wrkhtn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-wrkhtn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.wrkhtn\nDeploy daemon haproxy.nfs.foo.smithi193.wrkhtn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.051336Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gsgxzb on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gsgxzb\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gsgxzb\nDeploy daemon haproxy.nfs.foo.smithi097.gsgxzb ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T06:59:39.053179Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.cysbma on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:46.438 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:46.965 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:46 smithi097 bash[24958]: cluster 2023-11-17T07:00:45.375075+0000 mgr.smithi097.swvonl (mgr.14180) 515 : cluster [DBG] pgmap v406: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:46.966 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:46 smithi097 bash[24958]: audit 2023-11-17T07:00:45.557609+0000 mgr.smithi097.swvonl (mgr.14180) 516 : audit [DBG] from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:47.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:46 smithi193 bash[28559]: cluster 2023-11-17T07:00:45.375075+0000 mgr.smithi097.swvonl (mgr.14180) 515 : cluster [DBG] pgmap v406: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:00:47.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:46 smithi193 bash[28559]: audit 2023-11-17T07:00:45.557609+0000 mgr.smithi097.swvonl (mgr.14180) 516 : audit [DBG] from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:47.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:48.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:47 smithi097 bash[24958]: audit 2023-11-17T07:00:46.953845+0000 mon.smithi097 (mon.0) 793 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ptseaz"}]: dispatch 2023-11-17T07:00:48.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:47 smithi193 bash[28559]: audit 2023-11-17T07:00:46.953845+0000 mon.smithi097 (mon.0) 793 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ptseaz"}]: dispatch 2023-11-17T07:00:49.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cephadm 2023-11-17T07:00:46.953008+0000 mgr.smithi097.swvonl (mgr.14180) 517 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cephadm 2023-11-17T07:00:46.953288+0000 mgr.smithi097.swvonl (mgr.14180) 518 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cephadm 2023-11-17T07:00:46.955412+0000 mgr.smithi097.swvonl (mgr.14180) 519 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cephadm 2023-11-17T07:00:46.959275+0000 mgr.smithi097.swvonl (mgr.14180) 520 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:49.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cephadm 2023-11-17T07:00:46.962451+0000 mgr.smithi097.swvonl (mgr.14180) 521 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:49.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cluster 2023-11-17T07:00:46.963830+0000 mgr.smithi097.swvonl (mgr.14180) 522 : cluster [DBG] pgmap v407: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:00:49.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: cluster 2023-11-17T07:00:47.711797+0000 mon.smithi097 (mon.0) 794 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:00:49.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:48 smithi097 bash[24958]: audit 2023-11-17T07:00:48.531769+0000 mon.smithi097 (mon.0) 795 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cephadm 2023-11-17T07:00:46.953008+0000 mgr.smithi097.swvonl (mgr.14180) 517 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:00:49.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cephadm 2023-11-17T07:00:46.953288+0000 mgr.smithi097.swvonl (mgr.14180) 518 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cephadm 2023-11-17T07:00:46.955412+0000 mgr.smithi097.swvonl (mgr.14180) 519 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:00:49.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ptseaz ... 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:00:49.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:00:49.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cephadm 2023-11-17T07:00:46.959275+0000 mgr.smithi097.swvonl (mgr.14180) 520 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:49.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cephadm 2023-11-17T07:00:46.962451+0000 mgr.smithi097.swvonl (mgr.14180) 521 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:00:49.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cluster 2023-11-17T07:00:46.963830+0000 mgr.smithi097.swvonl (mgr.14180) 522 : cluster [DBG] pgmap v407: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:00:49.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: cluster 2023-11-17T07:00:47.711797+0000 mon.smithi097 (mon.0) 794 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:00:49.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:48 smithi193 bash[28559]: audit 2023-11-17T07:00:48.531769+0000 mon.smithi097 (mon.0) 795 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:51.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:50 smithi097 bash[24958]: cluster 2023-11-17T07:00:48.964920+0000 mgr.smithi097.swvonl (mgr.14180) 523 : cluster [DBG] pgmap v408: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:00:51.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:50 smithi193 bash[28559]: cluster 2023-11-17T07:00:48.964920+0000 mgr.smithi097.swvonl (mgr.14180) 523 : cluster [DBG] pgmap v408: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:00:51.154 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:51.154 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:52.037 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:53.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:53.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:52 smithi097 bash[24958]: cluster 2023-11-17T07:00:50.966301+0000 mgr.smithi097.swvonl (mgr.14180) 524 : cluster [DBG] pgmap v409: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:00:53.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:52 smithi097 bash[24958]: audit 2023-11-17T07:00:51.140565+0000 mgr.smithi097.swvonl (mgr.14180) 525 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:53.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:52 smithi193 bash[28559]: cluster 2023-11-17T07:00:50.966301+0000 mgr.smithi097.swvonl (mgr.14180) 524 : cluster [DBG] pgmap v409: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:00:53.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:52 smithi193 bash[28559]: audit 2023-11-17T07:00:51.140565+0000 mgr.smithi097.swvonl (mgr.14180) 525 : audit [DBG] from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:55.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:54 smithi097 bash[24958]: cluster 2023-11-17T07:00:52.967240+0000 mgr.smithi097.swvonl (mgr.14180) 526 : cluster [DBG] pgmap v410: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:00:55.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:54 smithi193 bash[28559]: cluster 2023-11-17T07:00:52.967240+0000 mgr.smithi097.swvonl (mgr.14180) 526 : cluster [DBG] pgmap v410: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:00:56.670 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:00:56.670 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:00:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:56 smithi097 bash[24958]: cluster 2023-11-17T07:00:54.968543+0000 mgr.smithi097.swvonl (mgr.14180) 527 : cluster [DBG] pgmap v411: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:00:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:56 smithi097 bash[24958]: audit 2023-11-17T07:00:56.520283+0000 mon.smithi097 (mon.0) 796 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:56 smithi193 bash[28559]: cluster 2023-11-17T07:00:54.968543+0000 mgr.smithi097.swvonl (mgr.14180) 527 : cluster [DBG] pgmap v411: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:00:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:56 smithi193 bash[28559]: audit 2023-11-17T07:00:56.520283+0000 mon.smithi097 (mon.0) 796 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:00:57.543 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:00:57.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:57 smithi097 bash[24958]: audit 2023-11-17T07:00:56.658995+0000 mgr.smithi097.swvonl (mgr.14180) 528 : audit [DBG] from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:58.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:57 smithi193 bash[28559]: audit 2023-11-17T07:00:56.658995+0000 mgr.smithi097.swvonl (mgr.14180) 528 : audit [DBG] from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:00:58.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:00:59.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:00:58 smithi097 bash[24958]: cluster 2023-11-17T07:00:56.969462+0000 mgr.smithi097.swvonl (mgr.14180) 529 : cluster [DBG] pgmap v412: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:00:59.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:00:58 smithi193 bash[28559]: cluster 2023-11-17T07:00:56.969462+0000 mgr.smithi097.swvonl (mgr.14180) 529 : cluster [DBG] pgmap v412: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:01:01.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:00 smithi097 bash[24958]: cluster 2023-11-17T07:00:58.970232+0000 mgr.smithi097.swvonl (mgr.14180) 530 : cluster [DBG] pgmap v413: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:01.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:00 smithi193 bash[28559]: cluster 2023-11-17T07:00:58.970232+0000 mgr.smithi097.swvonl (mgr.14180) 530 : cluster [DBG] pgmap v413: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:02.189 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:01:02.189 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:01:02.963 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:01:03.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:02 smithi097 bash[24958]: cluster 2023-11-17T07:01:00.971508+0000 mgr.smithi097.swvonl (mgr.14180) 531 : cluster [DBG] pgmap v414: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:01:03.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:02 smithi193 bash[28559]: cluster 2023-11-17T07:01:00.971508+0000 mgr.smithi097.swvonl (mgr.14180) 531 : cluster [DBG] pgmap v414: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:01:03.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:01:04.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:03 smithi097 bash[24958]: audit 2023-11-17T07:01:02.180779+0000 mgr.smithi097.swvonl (mgr.14180) 532 : audit [DBG] from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:04.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:03 smithi193 bash[28559]: audit 2023-11-17T07:01:02.180779+0000 mgr.smithi097.swvonl (mgr.14180) 532 : audit [DBG] from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:05.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:04 smithi097 bash[24958]: cluster 2023-11-17T07:01:02.972459+0000 mgr.smithi097.swvonl (mgr.14180) 533 : cluster [DBG] pgmap v415: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:05.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:04 smithi193 bash[28559]: cluster 2023-11-17T07:01:02.972459+0000 mgr.smithi097.swvonl (mgr.14180) 533 : cluster [DBG] pgmap v415: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:05.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:05 smithi097 bash[24958]: cluster 2023-11-17T07:01:04.973823+0000 mgr.smithi097.swvonl (mgr.14180) 534 : cluster [DBG] pgmap v416: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:06.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:05 smithi193 bash[28559]: cluster 2023-11-17T07:01:04.973823+0000 mgr.smithi097.swvonl (mgr.14180) 534 : cluster [DBG] pgmap v416: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:07.507 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:01:07.508 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:01:08.326 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:01:08.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:08 smithi097 bash[24958]: cluster 2023-11-17T07:01:06.974883+0000 mgr.smithi097.swvonl (mgr.14180) 535 : cluster [DBG] pgmap v417: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:08.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:08 smithi097 bash[24958]: audit 2023-11-17T07:01:07.494481+0000 mgr.smithi097.swvonl (mgr.14180) 536 : audit [DBG] from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:08.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:08 smithi193 bash[28559]: cluster 2023-11-17T07:01:06.974883+0000 mgr.smithi097.swvonl (mgr.14180) 535 : cluster [DBG] pgmap v417: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:08.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:08 smithi193 bash[28559]: audit 2023-11-17T07:01:07.494481+0000 mgr.smithi097.swvonl (mgr.14180) 536 : audit [DBG] from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:09.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:01:10.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:10 smithi097 bash[24958]: cluster 2023-11-17T07:01:08.975976+0000 mgr.smithi097.swvonl (mgr.14180) 537 : cluster [DBG] pgmap v418: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:10.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:10 smithi193 bash[28559]: cluster 2023-11-17T07:01:08.975976+0000 mgr.smithi097.swvonl (mgr.14180) 537 : cluster [DBG] pgmap v418: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:12.542 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:12 smithi097 bash[24958]: cluster 2023-11-17T07:01:10.977266+0000 mgr.smithi097.swvonl (mgr.14180) 538 : cluster [DBG] pgmap v419: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:01:12.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:12 smithi193 bash[28559]: cluster 2023-11-17T07:01:10.977266+0000 mgr.smithi097.swvonl (mgr.14180) 538 : cluster [DBG] pgmap v419: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:01:12.969 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:01:12.969 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:01:13.819 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:01:14.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:14 smithi097 bash[24958]: audit 2023-11-17T07:01:12.955567+0000 mgr.smithi097.swvonl (mgr.14180) 539 : audit [DBG] from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:14.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:14 smithi097 bash[24958]: cluster 2023-11-17T07:01:12.978153+0000 mgr.smithi097.swvonl (mgr.14180) 540 : cluster [DBG] pgmap v420: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:14.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:14 smithi193 bash[28559]: audit 2023-11-17T07:01:12.955567+0000 mgr.smithi097.swvonl (mgr.14180) 539 : audit [DBG] from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:01:14.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:14 smithi193 bash[28559]: cluster 2023-11-17T07:01:12.978153+0000 mgr.smithi097.swvonl (mgr.14180) 540 : cluster [DBG] pgmap v420: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:01:14.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:01:16.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:16 smithi097 bash[24958]: cluster 2023-11-17T07:01:14.979444+0000 mgr.smithi097.swvonl (mgr.14180) 541 : cluster [DBG] pgmap v421: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.555 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:01.555 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T06:59:39.054851Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.xrxejj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:43.158872Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.xphbfi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-xphbfi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.xphbfi\nDeploy daemon haproxy.nfs.foo.smithi193.xphbfi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.955200Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ptseaz on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ptseaz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ptseaz\nDeploy daemon haproxy.nfs.foo.smithi097.ptseaz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:00:46.959104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ouokaf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:01.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:18 smithi097 bash[24958]: cluster 2023-11-17T07:01:16.980451+0000 mgr.smithi097.swvonl (mgr.14180) 542 : cluster [DBG] pgmap v422: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.559 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:16 smithi193 bash[28559]: cluster 2023-11-17T07:01:14.979444+0000 mgr.smithi097.swvonl (mgr.14180) 541 : cluster [DBG] pgmap v421: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:18 smithi193 bash[28559]: cluster 2023-11-17T07:01:16.980451+0000 mgr.smithi097.swvonl (mgr.14180) 542 : cluster [DBG] pgmap v422: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:19 smithi193 bash[28559]: audit 2023-11-17T07:01:18.323337+0000 mgr.smithi097.swvonl (mgr.14180) 543 : audit [DBG] from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:20 smithi193 bash[28559]: cluster 2023-11-17T07:01:18.981473+0000 mgr.smithi097.swvonl (mgr.14180) 544 : cluster [DBG] pgmap v423: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:22 smithi193 bash[28559]: cluster 2023-11-17T07:01:20.982853+0000 mgr.smithi097.swvonl (mgr.14180) 545 : cluster [DBG] pgmap v424: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:24 smithi193 bash[28559]: cluster 2023-11-17T07:01:22.983840+0000 mgr.smithi097.swvonl (mgr.14180) 546 : cluster [DBG] pgmap v425: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.582 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:26 smithi193 bash[28559]: cluster 2023-11-17T07:01:24.985237+0000 mgr.smithi097.swvonl (mgr.14180) 547 : cluster [DBG] pgmap v426: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:28 smithi193 bash[28559]: cluster 2023-11-17T07:01:26.986320+0000 mgr.smithi097.swvonl (mgr.14180) 548 : cluster [DBG] pgmap v427: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:30 smithi193 bash[28559]: cluster 2023-11-17T07:01:28.987409+0000 mgr.smithi097.swvonl (mgr.14180) 549 : cluster [DBG] pgmap v428: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:32 smithi193 bash[28559]: cluster 2023-11-17T07:01:30.988784+0000 mgr.smithi097.swvonl (mgr.14180) 550 : cluster [DBG] pgmap v429: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:34 smithi193 bash[28559]: cluster 2023-11-17T07:01:32.989848+0000 mgr.smithi097.swvonl (mgr.14180) 551 : cluster [DBG] pgmap v430: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:36 smithi193 bash[28559]: cluster 2023-11-17T07:01:34.991274+0000 mgr.smithi097.swvonl (mgr.14180) 552 : cluster [DBG] pgmap v431: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:38 smithi193 bash[28559]: cluster 2023-11-17T07:01:36.992322+0000 mgr.smithi097.swvonl (mgr.14180) 553 : cluster [DBG] pgmap v432: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:40 smithi193 bash[28559]: cluster 2023-11-17T07:01:38.993391+0000 mgr.smithi097.swvonl (mgr.14180) 554 : cluster [DBG] pgmap v433: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:42 smithi193 bash[28559]: cluster 2023-11-17T07:01:40.994650+0000 mgr.smithi097.swvonl (mgr.14180) 555 : cluster [DBG] pgmap v434: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:44 smithi193 bash[28559]: cluster 2023-11-17T07:01:42.995644+0000 mgr.smithi097.swvonl (mgr.14180) 556 : cluster [DBG] pgmap v435: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:46 smithi193 bash[28559]: cluster 2023-11-17T07:01:44.997002+0000 mgr.smithi097.swvonl (mgr.14180) 557 : cluster [DBG] pgmap v436: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:46.967521+0000 mon.smithi097 (mon.0) 797 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:47.266203+0000 mon.smithi097 (mon.0) 798 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:02:01.583 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:47.267551+0000 mon.smithi097 (mon.0) 799 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:47.275554+0000 mon.smithi097 (mon.0) 800 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:47.282053+0000 mon.smithi097 (mon.0) 801 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:47 smithi193 bash[28559]: audit 2023-11-17T07:01:47.296088+0000 mon.smithi097 (mon.0) 802 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:48 smithi193 bash[28559]: cluster 2023-11-17T07:01:46.997906+0000 mgr.smithi097.swvonl (mgr.14180) 558 : cluster [DBG] pgmap v437: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:48 smithi193 bash[28559]: cluster 2023-11-17T07:01:47.276375+0000 mgr.smithi097.swvonl (mgr.14180) 559 : cluster [DBG] pgmap v438: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:48 smithi193 bash[28559]: cephadm 2023-11-17T07:01:47.300195+0000 mgr.smithi097.swvonl (mgr.14180) 560 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.zechgt on smithi193 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:48 smithi193 bash[28559]: cluster 2023-11-17T07:01:47.379741+0000 mon.smithi097 (mon.0) 803 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:48 smithi193 bash[28559]: cluster 2023-11-17T07:01:47.379791+0000 mon.smithi097 (mon.0) 804 : cluster [INF] Cluster is now healthy 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:50 smithi193 bash[28559]: cluster 2023-11-17T07:01:49.277182+0000 mgr.smithi097.swvonl (mgr.14180) 561 : cluster [DBG] pgmap v439: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:51 smithi193 bash[28559]: audit 2023-11-17T07:01:51.185455+0000 mon.smithi097 (mon.0) 805 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.zechgt"}]: dispatch 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: cephadm 2023-11-17T07:01:51.184387+0000 mgr.smithi097.swvonl (mgr.14180) 562 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.584 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:02:01.585 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.586 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: cephadm 2023-11-17T07:01:51.184889+0000 mgr.smithi097.swvonl (mgr.14180) 563 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: cephadm 2023-11-17T07:01:51.187166+0000 mgr.smithi097.swvonl (mgr.14180) 564 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: cephadm 2023-11-17T07:01:51.192195+0000 mgr.smithi097.swvonl (mgr.14180) 565 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.sbdqju on smithi097 2023-11-17T07:02:01.587 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:52 smithi193 bash[28559]: cluster 2023-11-17T07:01:51.278080+0000 mgr.smithi097.swvonl (mgr.14180) 566 : cluster [DBG] pgmap v440: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:54 smithi193 bash[28559]: cluster 2023-11-17T07:01:53.278859+0000 mgr.smithi097.swvonl (mgr.14180) 567 : cluster [DBG] pgmap v441: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:55 smithi193 bash[28559]: audit 2023-11-17T07:01:54.932760+0000 mon.smithi097 (mon.0) 806 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.sbdqju"}]: dispatch 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cephadm 2023-11-17T07:01:54.931662+0000 mgr.smithi097.swvonl (mgr.14180) 568 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.588 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.589 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cephadm 2023-11-17T07:01:54.932085+0000 mgr.smithi097.swvonl (mgr.14180) 569 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cephadm 2023-11-17T07:01:54.934328+0000 mgr.smithi097.swvonl (mgr.14180) 570 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.590 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.591 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:02:01.591 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.591 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cephadm 2023-11-17T07:01:54.937447+0000 mgr.smithi097.swvonl (mgr.14180) 571 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cephadm 2023-11-17T07:01:54.940309+0000 mgr.smithi097.swvonl (mgr.14180) 572 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cluster 2023-11-17T07:01:54.941685+0000 mgr.smithi097.swvonl (mgr.14180) 573 : cluster [DBG] pgmap v442: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:56 smithi193 bash[28559]: cluster 2023-11-17T07:01:55.410988+0000 mon.smithi097 (mon.0) 807 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:58 smithi193 bash[28559]: cluster 2023-11-17T07:01:56.942742+0000 mgr.smithi097.swvonl (mgr.14180) 574 : cluster [DBG] pgmap v443: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:01:59 smithi193 bash[28559]: audit 2023-11-17T07:01:58.585350+0000 mon.smithi097 (mon.0) 808 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.592 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:00 smithi193 bash[28559]: cluster 2023-11-17T07:01:58.943819+0000 mgr.smithi097.swvonl (mgr.14180) 575 : cluster [DBG] pgmap v444: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:01.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:19 smithi097 bash[24958]: audit 2023-11-17T07:01:18.323337+0000 mgr.smithi097.swvonl (mgr.14180) 543 : audit [DBG] from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:01.593 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:20 smithi097 bash[24958]: cluster 2023-11-17T07:01:18.981473+0000 mgr.smithi097.swvonl (mgr.14180) 544 : cluster [DBG] pgmap v423: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:22 smithi097 bash[24958]: cluster 2023-11-17T07:01:20.982853+0000 mgr.smithi097.swvonl (mgr.14180) 545 : cluster [DBG] pgmap v424: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:24 smithi097 bash[24958]: cluster 2023-11-17T07:01:22.983840+0000 mgr.smithi097.swvonl (mgr.14180) 546 : cluster [DBG] pgmap v425: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:26 smithi097 bash[24958]: cluster 2023-11-17T07:01:24.985237+0000 mgr.smithi097.swvonl (mgr.14180) 547 : cluster [DBG] pgmap v426: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:28 smithi097 bash[24958]: cluster 2023-11-17T07:01:26.986320+0000 mgr.smithi097.swvonl (mgr.14180) 548 : cluster [DBG] pgmap v427: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:30 smithi097 bash[24958]: cluster 2023-11-17T07:01:28.987409+0000 mgr.smithi097.swvonl (mgr.14180) 549 : cluster [DBG] pgmap v428: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:32 smithi097 bash[24958]: cluster 2023-11-17T07:01:30.988784+0000 mgr.smithi097.swvonl (mgr.14180) 550 : cluster [DBG] pgmap v429: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:34 smithi097 bash[24958]: cluster 2023-11-17T07:01:32.989848+0000 mgr.smithi097.swvonl (mgr.14180) 551 : cluster [DBG] pgmap v430: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:36 smithi097 bash[24958]: cluster 2023-11-17T07:01:34.991274+0000 mgr.smithi097.swvonl (mgr.14180) 552 : cluster [DBG] pgmap v431: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:38 smithi097 bash[24958]: cluster 2023-11-17T07:01:36.992322+0000 mgr.smithi097.swvonl (mgr.14180) 553 : cluster [DBG] pgmap v432: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.594 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:40 smithi097 bash[24958]: cluster 2023-11-17T07:01:38.993391+0000 mgr.smithi097.swvonl (mgr.14180) 554 : cluster [DBG] pgmap v433: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:42 smithi097 bash[24958]: cluster 2023-11-17T07:01:40.994650+0000 mgr.smithi097.swvonl (mgr.14180) 555 : cluster [DBG] pgmap v434: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:44 smithi097 bash[24958]: cluster 2023-11-17T07:01:42.995644+0000 mgr.smithi097.swvonl (mgr.14180) 556 : cluster [DBG] pgmap v435: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:46 smithi097 bash[24958]: cluster 2023-11-17T07:01:44.997002+0000 mgr.smithi097.swvonl (mgr.14180) 557 : cluster [DBG] pgmap v436: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:46.967521+0000 mon.smithi097 (mon.0) 797 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:47.266203+0000 mon.smithi097 (mon.0) 798 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:47.267551+0000 mon.smithi097 (mon.0) 799 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:47.275554+0000 mon.smithi097 (mon.0) 800 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.595 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:47.282053+0000 mon.smithi097 (mon.0) 801 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:47 smithi097 bash[24958]: audit 2023-11-17T07:01:47.296088+0000 mon.smithi097 (mon.0) 802 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:48 smithi097 bash[24958]: cluster 2023-11-17T07:01:46.997906+0000 mgr.smithi097.swvonl (mgr.14180) 558 : cluster [DBG] pgmap v437: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:48 smithi097 bash[24958]: cluster 2023-11-17T07:01:47.276375+0000 mgr.smithi097.swvonl (mgr.14180) 559 : cluster [DBG] pgmap v438: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:48 smithi097 bash[24958]: cephadm 2023-11-17T07:01:47.300195+0000 mgr.smithi097.swvonl (mgr.14180) 560 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.zechgt on smithi193 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:48 smithi097 bash[24958]: cluster 2023-11-17T07:01:47.379741+0000 mon.smithi097 (mon.0) 803 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:48 smithi097 bash[24958]: cluster 2023-11-17T07:01:47.379791+0000 mon.smithi097 (mon.0) 804 : cluster [INF] Cluster is now healthy 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:50 smithi097 bash[24958]: cluster 2023-11-17T07:01:49.277182+0000 mgr.smithi097.swvonl (mgr.14180) 561 : cluster [DBG] pgmap v439: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:51 smithi097 bash[24958]: audit 2023-11-17T07:01:51.185455+0000 mon.smithi097 (mon.0) 805 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.zechgt"}]: dispatch 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: cephadm 2023-11-17T07:01:51.184387+0000 mgr.smithi097.swvonl (mgr.14180) 562 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.596 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:02:01.597 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: cephadm 2023-11-17T07:01:51.184889+0000 mgr.smithi097.swvonl (mgr.14180) 563 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: cephadm 2023-11-17T07:01:51.187166+0000 mgr.smithi097.swvonl (mgr.14180) 564 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.598 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zechgt ... 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: cephadm 2023-11-17T07:01:51.192195+0000 mgr.smithi097.swvonl (mgr.14180) 565 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.sbdqju on smithi097 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:52 smithi097 bash[24958]: cluster 2023-11-17T07:01:51.278080+0000 mgr.smithi097.swvonl (mgr.14180) 566 : cluster [DBG] pgmap v440: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:54 smithi097 bash[24958]: cluster 2023-11-17T07:01:53.278859+0000 mgr.smithi097.swvonl (mgr.14180) 567 : cluster [DBG] pgmap v441: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:02:01.599 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:55 smithi097 bash[24958]: audit 2023-11-17T07:01:54.932760+0000 mon.smithi097 (mon.0) 806 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.sbdqju"}]: dispatch 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cephadm 2023-11-17T07:01:54.931662+0000 mgr.smithi097.swvonl (mgr.14180) 568 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.600 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.601 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.602 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.602 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.603 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.603 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.603 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.604 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.604 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.604 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.605 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cephadm 2023-11-17T07:01:54.932085+0000 mgr.smithi097.swvonl (mgr.14180) 569 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.605 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cephadm 2023-11-17T07:01:54.934328+0000 mgr.smithi097.swvonl (mgr.14180) 570 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.605 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sbdqju ... 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:02:01.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cephadm 2023-11-17T07:01:54.937447+0000 mgr.smithi097.swvonl (mgr.14180) 571 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cephadm 2023-11-17T07:01:54.940309+0000 mgr.smithi097.swvonl (mgr.14180) 572 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cluster 2023-11-17T07:01:54.941685+0000 mgr.smithi097.swvonl (mgr.14180) 573 : cluster [DBG] pgmap v442: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:56 smithi097 bash[24958]: cluster 2023-11-17T07:01:55.410988+0000 mon.smithi097 (mon.0) 807 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:58 smithi097 bash[24958]: cluster 2023-11-17T07:01:56.942742+0000 mgr.smithi097.swvonl (mgr.14180) 574 : cluster [DBG] pgmap v443: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:01:59 smithi097 bash[24958]: audit 2023-11-17T07:01:58.585350+0000 mon.smithi097 (mon.0) 808 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:01.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:00 smithi097 bash[24958]: cluster 2023-11-17T07:01:58.943819+0000 mgr.smithi097.swvonl (mgr.14180) 575 : cluster [DBG] pgmap v444: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:02.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:03.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:02 smithi097 bash[24958]: cluster 2023-11-17T07:02:00.945226+0000 mgr.smithi097.swvonl (mgr.14180) 576 : cluster [DBG] pgmap v445: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2023-11-17T07:02:03.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:02 smithi193 bash[28559]: cluster 2023-11-17T07:02:00.945226+0000 mgr.smithi097.swvonl (mgr.14180) 576 : cluster [DBG] pgmap v445: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2023-11-17T07:02:05.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:04 smithi097 bash[24958]: cluster 2023-11-17T07:02:02.946002+0000 mgr.smithi097.swvonl (mgr.14180) 577 : cluster [DBG] pgmap v446: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:05.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:04 smithi193 bash[28559]: cluster 2023-11-17T07:02:02.946002+0000 mgr.smithi097.swvonl (mgr.14180) 577 : cluster [DBG] pgmap v446: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:06.290 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:06.291 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:06.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:06 smithi097 bash[24958]: cluster 2023-11-17T07:02:04.946882+0000 mgr.smithi097.swvonl (mgr.14180) 578 : cluster [DBG] pgmap v447: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:07.049 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:07.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:06 smithi193 bash[28559]: cluster 2023-11-17T07:02:04.946882+0000 mgr.smithi097.swvonl (mgr.14180) 578 : cluster [DBG] pgmap v447: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:02:08.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:07 smithi097 bash[24958]: audit 2023-11-17T07:02:06.282040+0000 mgr.smithi097.swvonl (mgr.14180) 579 : audit [DBG] from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:08.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:07 smithi193 bash[28559]: audit 2023-11-17T07:02:06.282040+0000 mgr.smithi097.swvonl (mgr.14180) 579 : audit [DBG] from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:09.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:08 smithi097 bash[24958]: cluster 2023-11-17T07:02:06.948032+0000 mgr.smithi097.swvonl (mgr.14180) 580 : cluster [DBG] pgmap v448: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:09.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:08 smithi193 bash[28559]: cluster 2023-11-17T07:02:06.948032+0000 mgr.smithi097.swvonl (mgr.14180) 580 : cluster [DBG] pgmap v448: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:11.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:10 smithi097 bash[24958]: cluster 2023-11-17T07:02:08.948644+0000 mgr.smithi097.swvonl (mgr.14180) 581 : cluster [DBG] pgmap v449: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:11.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:10 smithi193 bash[28559]: cluster 2023-11-17T07:02:08.948644+0000 mgr.smithi097.swvonl (mgr.14180) 581 : cluster [DBG] pgmap v449: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:11.748 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:11.748 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:12.619 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:13.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:12 smithi097 bash[24958]: cluster 2023-11-17T07:02:10.949984+0000 mgr.smithi097.swvonl (mgr.14180) 582 : cluster [DBG] pgmap v450: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:13.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:12 smithi097 bash[24958]: audit 2023-11-17T07:02:11.735546+0000 mgr.smithi097.swvonl (mgr.14180) 583 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:13.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:12 smithi193 bash[28559]: cluster 2023-11-17T07:02:10.949984+0000 mgr.smithi097.swvonl (mgr.14180) 582 : cluster [DBG] pgmap v450: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:13.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:12 smithi193 bash[28559]: audit 2023-11-17T07:02:11.735546+0000 mgr.smithi097.swvonl (mgr.14180) 583 : audit [DBG] from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:13.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:14.916 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:14 smithi097 bash[24958]: cluster 2023-11-17T07:02:12.951040+0000 mgr.smithi097.swvonl (mgr.14180) 584 : cluster [DBG] pgmap v451: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:14 smithi193 bash[28559]: cluster 2023-11-17T07:02:12.951040+0000 mgr.smithi097.swvonl (mgr.14180) 584 : cluster [DBG] pgmap v451: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:16.909 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:16 smithi097 bash[24958]: cluster 2023-11-17T07:02:14.952491+0000 mgr.smithi097.swvonl (mgr.14180) 585 : cluster [DBG] pgmap v452: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:16 smithi193 bash[28559]: cluster 2023-11-17T07:02:14.952491+0000 mgr.smithi097.swvonl (mgr.14180) 585 : cluster [DBG] pgmap v452: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:17.353 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:17.354 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:18.164 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:19.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:18 smithi097 bash[24958]: cluster 2023-11-17T07:02:16.953545+0000 mgr.smithi097.swvonl (mgr.14180) 586 : cluster [DBG] pgmap v453: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:19.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:18 smithi097 bash[24958]: audit 2023-11-17T07:02:17.342542+0000 mgr.smithi097.swvonl (mgr.14180) 587 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:18 smithi193 bash[28559]: cluster 2023-11-17T07:02:16.953545+0000 mgr.smithi097.swvonl (mgr.14180) 586 : cluster [DBG] pgmap v453: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:18 smithi193 bash[28559]: audit 2023-11-17T07:02:17.342542+0000 mgr.smithi097.swvonl (mgr.14180) 587 : audit [DBG] from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:19.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:20.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:20 smithi097 bash[24958]: cluster 2023-11-17T07:02:18.954683+0000 mgr.smithi097.swvonl (mgr.14180) 588 : cluster [DBG] pgmap v454: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:21.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:20 smithi193 bash[28559]: cluster 2023-11-17T07:02:18.954683+0000 mgr.smithi097.swvonl (mgr.14180) 588 : cluster [DBG] pgmap v454: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:22.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:22 smithi097 bash[24958]: cluster 2023-11-17T07:02:20.956038+0000 mgr.smithi097.swvonl (mgr.14180) 589 : cluster [DBG] pgmap v455: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:22.922 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:23.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:22 smithi193 bash[28559]: cluster 2023-11-17T07:02:20.956038+0000 mgr.smithi097.swvonl (mgr.14180) 589 : cluster [DBG] pgmap v455: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:23.806 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:24.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:25.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:24 smithi097 bash[24958]: audit 2023-11-17T07:02:22.907576+0000 mgr.smithi097.swvonl (mgr.14180) 590 : audit [DBG] from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:25.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:24 smithi097 bash[24958]: cluster 2023-11-17T07:02:22.956624+0000 mgr.smithi097.swvonl (mgr.14180) 591 : cluster [DBG] pgmap v456: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:24 smithi193 bash[28559]: audit 2023-11-17T07:02:22.907576+0000 mgr.smithi097.swvonl (mgr.14180) 590 : audit [DBG] from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:24 smithi193 bash[28559]: cluster 2023-11-17T07:02:22.956624+0000 mgr.smithi097.swvonl (mgr.14180) 591 : cluster [DBG] pgmap v456: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:26.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:26 smithi097 bash[24958]: cluster 2023-11-17T07:02:24.957966+0000 mgr.smithi097.swvonl (mgr.14180) 592 : cluster [DBG] pgmap v457: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:27.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:26 smithi193 bash[28559]: cluster 2023-11-17T07:02:24.957966+0000 mgr.smithi097.swvonl (mgr.14180) 592 : cluster [DBG] pgmap v457: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:28.507 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:28.507 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:28.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:28 smithi097 bash[24958]: cluster 2023-11-17T07:02:26.959052+0000 mgr.smithi097.swvonl (mgr.14180) 593 : cluster [DBG] pgmap v458: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:29.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:28 smithi193 bash[28559]: cluster 2023-11-17T07:02:26.959052+0000 mgr.smithi097.swvonl (mgr.14180) 593 : cluster [DBG] pgmap v458: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:29.306 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:30.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:29 smithi097 bash[24958]: audit 2023-11-17T07:02:28.492144+0000 mgr.smithi097.swvonl (mgr.14180) 594 : audit [DBG] from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:29 smithi193 bash[28559]: audit 2023-11-17T07:02:28.492144+0000 mgr.smithi097.swvonl (mgr.14180) 594 : audit [DBG] from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:30.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:30.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:30 smithi097 bash[24958]: cluster 2023-11-17T07:02:28.960186+0000 mgr.smithi097.swvonl (mgr.14180) 595 : cluster [DBG] pgmap v459: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:30 smithi193 bash[28559]: cluster 2023-11-17T07:02:28.960186+0000 mgr.smithi097.swvonl (mgr.14180) 595 : cluster [DBG] pgmap v459: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:33.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:32 smithi097 bash[24958]: cluster 2023-11-17T07:02:30.961579+0000 mgr.smithi097.swvonl (mgr.14180) 596 : cluster [DBG] pgmap v460: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:32 smithi193 bash[28559]: cluster 2023-11-17T07:02:30.961579+0000 mgr.smithi097.swvonl (mgr.14180) 596 : cluster [DBG] pgmap v460: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:34.009 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:34.010 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:34.815 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:34 smithi097 bash[24958]: cluster 2023-11-17T07:02:32.962568+0000 mgr.smithi097.swvonl (mgr.14180) 597 : cluster [DBG] pgmap v461: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:35.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:34 smithi193 bash[28559]: cluster 2023-11-17T07:02:32.962568+0000 mgr.smithi097.swvonl (mgr.14180) 597 : cluster [DBG] pgmap v461: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:35.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:36.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:35 smithi097 bash[24958]: audit 2023-11-17T07:02:33.993630+0000 mgr.smithi097.swvonl (mgr.14180) 598 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:36.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:35 smithi193 bash[28559]: audit 2023-11-17T07:02:33.993630+0000 mgr.smithi097.swvonl (mgr.14180) 598 : audit [DBG] from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:36.978 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:36 smithi097 bash[24958]: cluster 2023-11-17T07:02:34.963992+0000 mgr.smithi097.swvonl (mgr.14180) 599 : cluster [DBG] pgmap v462: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:36 smithi193 bash[28559]: cluster 2023-11-17T07:02:34.963992+0000 mgr.smithi097.swvonl (mgr.14180) 599 : cluster [DBG] pgmap v462: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:38.988 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:38 smithi097 bash[24958]: cluster 2023-11-17T07:02:36.965034+0000 mgr.smithi097.swvonl (mgr.14180) 600 : cluster [DBG] pgmap v463: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:39.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:38 smithi193 bash[28559]: cluster 2023-11-17T07:02:36.965034+0000 mgr.smithi097.swvonl (mgr.14180) 600 : cluster [DBG] pgmap v463: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:39.366 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:39.367 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:40.229 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:40 smithi097 bash[24958]: cluster 2023-11-17T07:02:38.966102+0000 mgr.smithi097.swvonl (mgr.14180) 601 : cluster [DBG] pgmap v464: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:40 smithi097 bash[24958]: audit 2023-11-17T07:02:39.353692+0000 mgr.smithi097.swvonl (mgr.14180) 602 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:40 smithi193 bash[28559]: cluster 2023-11-17T07:02:38.966102+0000 mgr.smithi097.swvonl (mgr.14180) 601 : cluster [DBG] pgmap v464: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:40 smithi193 bash[28559]: audit 2023-11-17T07:02:39.353692+0000 mgr.smithi097.swvonl (mgr.14180) 602 : audit [DBG] from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:41.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:43.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:42 smithi097 bash[24958]: cluster 2023-11-17T07:02:40.967468+0000 mgr.smithi097.swvonl (mgr.14180) 603 : cluster [DBG] pgmap v465: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:43.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:42 smithi193 bash[28559]: cluster 2023-11-17T07:02:40.967468+0000 mgr.smithi097.swvonl (mgr.14180) 603 : cluster [DBG] pgmap v465: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:44.853 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:44.853 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:45.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:44 smithi097 bash[24958]: cluster 2023-11-17T07:02:42.968494+0000 mgr.smithi097.swvonl (mgr.14180) 604 : cluster [DBG] pgmap v466: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:45.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:44 smithi193 bash[28559]: cluster 2023-11-17T07:02:42.968494+0000 mgr.smithi097.swvonl (mgr.14180) 604 : cluster [DBG] pgmap v466: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:45.739 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:46.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:47.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:46 smithi097 bash[24958]: audit 2023-11-17T07:02:44.839307+0000 mgr.smithi097.swvonl (mgr.14180) 605 : audit [DBG] from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:47.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:46 smithi097 bash[24958]: cluster 2023-11-17T07:02:44.969948+0000 mgr.smithi097.swvonl (mgr.14180) 606 : cluster [DBG] pgmap v467: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:47.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:46 smithi193 bash[28559]: audit 2023-11-17T07:02:44.839307+0000 mgr.smithi097.swvonl (mgr.14180) 605 : audit [DBG] from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:47.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:46 smithi193 bash[28559]: cluster 2023-11-17T07:02:44.969948+0000 mgr.smithi097.swvonl (mgr.14180) 606 : cluster [DBG] pgmap v467: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:48 smithi097 bash[24958]: cluster 2023-11-17T07:02:46.970700+0000 mgr.smithi097.swvonl (mgr.14180) 607 : cluster [DBG] pgmap v468: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:49.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:48 smithi193 bash[28559]: cluster 2023-11-17T07:02:46.970700+0000 mgr.smithi097.swvonl (mgr.14180) 607 : cluster [DBG] pgmap v468: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:50.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:50.280 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:50.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:50 smithi097 bash[24958]: cluster 2023-11-17T07:02:48.971581+0000 mgr.smithi097.swvonl (mgr.14180) 608 : cluster [DBG] pgmap v469: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:51.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:50 smithi193 bash[28559]: cluster 2023-11-17T07:02:48.971581+0000 mgr.smithi097.swvonl (mgr.14180) 608 : cluster [DBG] pgmap v469: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:51.096 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:52.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:51 smithi097 bash[24958]: audit 2023-11-17T07:02:50.271283+0000 mgr.smithi097.swvonl (mgr.14180) 609 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:52.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:51 smithi193 bash[28559]: audit 2023-11-17T07:02:50.271283+0000 mgr.smithi097.swvonl (mgr.14180) 609 : audit [DBG] from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:52.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:53.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:52 smithi097 bash[24958]: cluster 2023-11-17T07:02:50.972889+0000 mgr.smithi097.swvonl (mgr.14180) 610 : cluster [DBG] pgmap v470: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:53.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:52 smithi193 bash[28559]: cluster 2023-11-17T07:02:50.972889+0000 mgr.smithi097.swvonl (mgr.14180) 610 : cluster [DBG] pgmap v470: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:02:55.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:54 smithi097 bash[24958]: cluster 2023-11-17T07:02:52.973917+0000 mgr.smithi097.swvonl (mgr.14180) 611 : cluster [DBG] pgmap v471: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:55.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:54 smithi193 bash[28559]: cluster 2023-11-17T07:02:52.973917+0000 mgr.smithi097.swvonl (mgr.14180) 611 : cluster [DBG] pgmap v471: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:55.678 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:02:55.679 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:00:46.962289Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.htctyb on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:54.944899+0000 mon.smithi097 (mon.0) 809 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.231269+0000 mon.smithi097 (mon.0) 810 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.232650+0000 mon.smithi097 (mon.0) 811 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.233843+0000 mon.smithi097 (mon.0) 812 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.235044+0000 mon.smithi097 (mon.0) 813 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.242070+0000 mon.smithi097 (mon.0) 814 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.246573+0000 mon.smithi097 (mon.0) 815 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.247364+0000 mon.smithi097 (mon.0) 816 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.248167+0000 mon.smithi097 (mon.0) 817 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.248931+0000 mon.smithi097 (mon.0) 818 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.255068+0000 mon.smithi097 (mon.0) 819 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.256611+0000 mon.smithi097 (mon.0) 820 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.257696+0000 mon.smithi097 (mon.0) 821 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.263531+0000 mon.smithi097 (mon.0) 822 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.269726+0000 mon.smithi097 (mon.0) 823 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:02:56.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:55 smithi097 bash[24958]: audit 2023-11-17T07:02:55.283495+0000 mon.smithi097 (mon.0) 824 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:54.944899+0000 mon.smithi097 (mon.0) 809 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:02:56.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.231269+0000 mon.smithi097 (mon.0) 810 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.232650+0000 mon.smithi097 (mon.0) 811 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.233843+0000 mon.smithi097 (mon.0) 812 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.235044+0000 mon.smithi097 (mon.0) 813 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.242070+0000 mon.smithi097 (mon.0) 814 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.246573+0000 mon.smithi097 (mon.0) 815 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.247364+0000 mon.smithi097 (mon.0) 816 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.248167+0000 mon.smithi097 (mon.0) 817 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.248931+0000 mon.smithi097 (mon.0) 818 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.255068+0000 mon.smithi097 (mon.0) 819 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.256611+0000 mon.smithi097 (mon.0) 820 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:02:56.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.257696+0000 mon.smithi097 (mon.0) 821 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:02:56.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.263531+0000 mon.smithi097 (mon.0) 822 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.269726+0000 mon.smithi097 (mon.0) 823 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:02:56.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:55 smithi193 bash[28559]: audit 2023-11-17T07:02:55.283495+0000 mon.smithi097 (mon.0) 824 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:56.533 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cluster 2023-11-17T07:02:54.975081+0000 mgr.smithi097.swvonl (mgr.14180) 612 : cluster [DBG] pgmap v472: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cephadm 2023-11-17T07:02:55.235891+0000 mgr.smithi097.swvonl (mgr.14180) 613 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 2748M 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cephadm 2023-11-17T07:02:55.249642+0000 mgr.smithi097.swvonl (mgr.14180) 614 : cephadm [INF] Adjusting osd_memory_target on smithi097 to 1980M 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cluster 2023-11-17T07:02:55.264210+0000 mgr.smithi097.swvonl (mgr.14180) 615 : cluster [DBG] pgmap v473: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cephadm 2023-11-17T07:02:55.290711+0000 mgr.smithi097.swvonl (mgr.14180) 616 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.utofbr on smithi193 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: audit 2023-11-17T07:02:55.665151+0000 mgr.smithi097.swvonl (mgr.14180) 617 : audit [DBG] from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cluster 2023-11-17T07:02:55.774863+0000 mon.smithi097 (mon.0) 825 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:02:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: cluster 2023-11-17T07:02:55.774926+0000 mon.smithi097 (mon.0) 826 : cluster [INF] Cluster is now healthy 2023-11-17T07:02:57.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:56 smithi097 bash[24958]: audit 2023-11-17T07:02:56.522133+0000 mon.smithi097 (mon.0) 827 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cluster 2023-11-17T07:02:54.975081+0000 mgr.smithi097.swvonl (mgr.14180) 612 : cluster [DBG] pgmap v472: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cephadm 2023-11-17T07:02:55.235891+0000 mgr.smithi097.swvonl (mgr.14180) 613 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 2748M 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cephadm 2023-11-17T07:02:55.249642+0000 mgr.smithi097.swvonl (mgr.14180) 614 : cephadm [INF] Adjusting osd_memory_target on smithi097 to 1980M 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cluster 2023-11-17T07:02:55.264210+0000 mgr.smithi097.swvonl (mgr.14180) 615 : cluster [DBG] pgmap v473: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cephadm 2023-11-17T07:02:55.290711+0000 mgr.smithi097.swvonl (mgr.14180) 616 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.utofbr on smithi193 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: audit 2023-11-17T07:02:55.665151+0000 mgr.smithi097.swvonl (mgr.14180) 617 : audit [DBG] from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:02:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cluster 2023-11-17T07:02:55.774863+0000 mon.smithi097 (mon.0) 825 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:02:57.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: cluster 2023-11-17T07:02:55.774926+0000 mon.smithi097 (mon.0) 826 : cluster [INF] Cluster is now healthy 2023-11-17T07:02:57.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:56 smithi193 bash[28559]: audit 2023-11-17T07:02:56.522133+0000 mon.smithi097 (mon.0) 827 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:02:57.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:02:58.964 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:58 smithi193 bash[28559]: cluster 2023-11-17T07:02:57.264989+0000 mgr.smithi097.swvonl (mgr.14180) 618 : cluster [DBG] pgmap v474: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:02:59.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:58 smithi097 bash[24958]: cluster 2023-11-17T07:02:57.264989+0000 mgr.smithi097.swvonl (mgr.14180) 618 : cluster [DBG] pgmap v474: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:03:00.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:02:59 smithi193 bash[28559]: audit 2023-11-17T07:02:59.021641+0000 mon.smithi097 (mon.0) 828 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.utofbr"}]: dispatch 2023-11-17T07:03:00.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:02:59 smithi097 bash[24958]: audit 2023-11-17T07:02:59.021641+0000 mon.smithi097 (mon.0) 828 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.utofbr"}]: dispatch 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: cephadm 2023-11-17T07:02:59.020435+0000 mgr.smithi097.swvonl (mgr.14180) 619 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: cephadm 2023-11-17T07:02:59.021027+0000 mgr.smithi097.swvonl (mgr.14180) 620 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: cephadm 2023-11-17T07:02:59.023116+0000 mgr.smithi097.swvonl (mgr.14180) 621 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.491 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.492 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.492 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.492 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.492 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: cephadm 2023-11-17T07:02:59.028001+0000 mgr.smithi097.swvonl (mgr.14180) 622 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pmshap on smithi097 2023-11-17T07:03:01.492 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:00 smithi097 bash[24958]: cluster 2023-11-17T07:02:59.265581+0000 mgr.smithi097.swvonl (mgr.14180) 623 : cluster [DBG] pgmap v475: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:03:01.493 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:01.493 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:51.186945Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zechgt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zechgt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zechgt\nDeploy daemon haproxy.nfs.foo.smithi193.zechgt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.934084Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sbdqju on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sbdqju\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sbdqju\nDeploy daemon haproxy.nfs.foo.smithi097.sbdqju ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:01:54.937292Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.oehmfp on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: cephadm 2023-11-17T07:02:59.020435+0000 mgr.smithi097.swvonl (mgr.14180) 619 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:03:01.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: cephadm 2023-11-17T07:02:59.021027+0000 mgr.smithi097.swvonl (mgr.14180) 620 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: cephadm 2023-11-17T07:02:59.023116+0000 mgr.smithi097.swvonl (mgr.14180) 621 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.utofbr ... 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:01.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:01.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: cephadm 2023-11-17T07:02:59.028001+0000 mgr.smithi097.swvonl (mgr.14180) 622 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pmshap on smithi097 2023-11-17T07:03:01.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:00 smithi193 bash[28559]: cluster 2023-11-17T07:02:59.265581+0000 mgr.smithi097.swvonl (mgr.14180) 623 : cluster [DBG] pgmap v475: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:03:02.126 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:02.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:02 smithi097 bash[24958]: cluster 2023-11-17T07:03:01.266479+0000 mgr.smithi097.swvonl (mgr.14180) 624 : cluster [DBG] pgmap v476: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:03:02.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:02 smithi097 bash[24958]: audit 2023-11-17T07:03:01.351284+0000 mgr.smithi097.swvonl (mgr.14180) 625 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:02.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:02 smithi193 bash[28559]: cluster 2023-11-17T07:03:01.266479+0000 mgr.smithi097.swvonl (mgr.14180) 624 : cluster [DBG] pgmap v476: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:03:02.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:02 smithi193 bash[28559]: audit 2023-11-17T07:03:01.351284+0000 mgr.smithi097.swvonl (mgr.14180) 625 : audit [DBG] from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:03.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:03.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:03 smithi097 bash[24958]: audit 2023-11-17T07:03:03.073608+0000 mon.smithi097 (mon.0) 829 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pmshap"}]: dispatch 2023-11-17T07:03:03.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:03 smithi193 bash[28559]: audit 2023-11-17T07:03:03.073608+0000 mon.smithi097 (mon.0) 829 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pmshap"}]: dispatch 2023-11-17T07:03:04.584 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cephadm 2023-11-17T07:03:03.072674+0000 mgr.smithi097.swvonl (mgr.14180) 626 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.584 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:03:04.585 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.586 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cephadm 2023-11-17T07:03:03.073137+0000 mgr.smithi097.swvonl (mgr.14180) 627 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cephadm 2023-11-17T07:03:03.075009+0000 mgr.smithi097.swvonl (mgr.14180) 628 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.587 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cephadm 2023-11-17T07:03:03.079167+0000 mgr.smithi097.swvonl (mgr.14180) 629 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cephadm 2023-11-17T07:03:03.082642+0000 mgr.smithi097.swvonl (mgr.14180) 630 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cluster 2023-11-17T07:03:03.083432+0000 mgr.smithi097.swvonl (mgr.14180) 631 : cluster [DBG] pgmap v477: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: cluster 2023-11-17T07:03:03.291342+0000 mon.smithi097 (mon.0) 830 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:03:04.588 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:04 smithi097 bash[24958]: audit 2023-11-17T07:03:03.628758+0000 mon.smithi097 (mon.0) 831 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:03:04.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cephadm 2023-11-17T07:03:03.072674+0000 mgr.smithi097.swvonl (mgr.14180) 626 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cephadm 2023-11-17T07:03:03.073137+0000 mgr.smithi097.swvonl (mgr.14180) 627 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cephadm 2023-11-17T07:03:03.075009+0000 mgr.smithi097.swvonl (mgr.14180) 628 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pmshap ... 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:03:04.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cephadm 2023-11-17T07:03:03.079167+0000 mgr.smithi097.swvonl (mgr.14180) 629 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cephadm 2023-11-17T07:03:03.082642+0000 mgr.smithi097.swvonl (mgr.14180) 630 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cluster 2023-11-17T07:03:03.083432+0000 mgr.smithi097.swvonl (mgr.14180) 631 : cluster [DBG] pgmap v477: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: cluster 2023-11-17T07:03:03.291342+0000 mon.smithi097 (mon.0) 830 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:03:04.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:04 smithi193 bash[28559]: audit 2023-11-17T07:03:03.628758+0000 mon.smithi097 (mon.0) 831 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:03:06.747 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:12.143 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:12.146 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:06 smithi097 bash[24958]: cluster 2023-11-17T07:03:05.084836+0000 mgr.smithi097.swvonl (mgr.14180) 632 : cluster [DBG] pgmap v478: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:03:12.146 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:07 smithi097 bash[24958]: audit 2023-11-17T07:03:06.733412+0000 mgr.smithi097.swvonl (mgr.14180) 633 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:12.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:06 smithi193 bash[28559]: cluster 2023-11-17T07:03:05.084836+0000 mgr.smithi097.swvonl (mgr.14180) 632 : cluster [DBG] pgmap v478: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:03:12.146 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:07 smithi193 bash[28559]: audit 2023-11-17T07:03:06.733412+0000 mgr.smithi097.swvonl (mgr.14180) 633 : audit [DBG] from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:12.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:08 smithi193 bash[28559]: cluster 2023-11-17T07:03:07.085936+0000 mgr.smithi097.swvonl (mgr.14180) 634 : cluster [DBG] pgmap v479: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:12.147 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:10 smithi193 bash[28559]: cluster 2023-11-17T07:03:09.087081+0000 mgr.smithi097.swvonl (mgr.14180) 635 : cluster [DBG] pgmap v480: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:12.148 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:12.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:08 smithi097 bash[24958]: cluster 2023-11-17T07:03:07.085936+0000 mgr.smithi097.swvonl (mgr.14180) 634 : cluster [DBG] pgmap v479: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:12.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:10 smithi097 bash[24958]: cluster 2023-11-17T07:03:09.087081+0000 mgr.smithi097.swvonl (mgr.14180) 635 : cluster [DBG] pgmap v480: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:12.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:12 smithi097 bash[24958]: cluster 2023-11-17T07:03:11.088442+0000 mgr.smithi097.swvonl (mgr.14180) 636 : cluster [DBG] pgmap v481: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2023-11-17T07:03:12.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:12 smithi097 bash[24958]: audit 2023-11-17T07:03:11.522312+0000 mon.smithi097 (mon.0) 832 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:03:12.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:12 smithi193 bash[28559]: cluster 2023-11-17T07:03:11.088442+0000 mgr.smithi097.swvonl (mgr.14180) 636 : cluster [DBG] pgmap v481: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2023-11-17T07:03:12.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:12 smithi193 bash[28559]: audit 2023-11-17T07:03:11.522312+0000 mon.smithi097 (mon.0) 832 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:03:13.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:14 smithi097 bash[24958]: cluster 2023-11-17T07:03:13.089404+0000 mgr.smithi097.swvonl (mgr.14180) 637 : cluster [DBG] pgmap v482: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:14.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:14 smithi193 bash[28559]: cluster 2023-11-17T07:03:13.089404+0000 mgr.smithi097.swvonl (mgr.14180) 637 : cluster [DBG] pgmap v482: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:03:16.795 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:16.795 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:16.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:16 smithi097 bash[24958]: cluster 2023-11-17T07:03:15.090198+0000 mgr.smithi097.swvonl (mgr.14180) 638 : cluster [DBG] pgmap v483: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:16.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:16 smithi193 bash[28559]: cluster 2023-11-17T07:03:15.090198+0000 mgr.smithi097.swvonl (mgr.14180) 638 : cluster [DBG] pgmap v483: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:17.644 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:17.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:17 smithi097 bash[24958]: audit 2023-11-17T07:03:16.786173+0000 mgr.smithi097.swvonl (mgr.14180) 639 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:17.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:17 smithi193 bash[28559]: audit 2023-11-17T07:03:16.786173+0000 mgr.smithi097.swvonl (mgr.14180) 639 : audit [DBG] from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:18.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:18 smithi097 bash[24958]: cluster 2023-11-17T07:03:17.090790+0000 mgr.smithi097.swvonl (mgr.14180) 640 : cluster [DBG] pgmap v484: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:18.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:18 smithi193 bash[28559]: cluster 2023-11-17T07:03:17.090790+0000 mgr.smithi097.swvonl (mgr.14180) 640 : cluster [DBG] pgmap v484: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:20 smithi193 bash[28559]: cluster 2023-11-17T07:03:19.091454+0000 mgr.smithi097.swvonl (mgr.14180) 641 : cluster [DBG] pgmap v485: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:20 smithi097 bash[24958]: cluster 2023-11-17T07:03:19.091454+0000 mgr.smithi097.swvonl (mgr.14180) 641 : cluster [DBG] pgmap v485: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:23.861 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:22 smithi193 bash[28559]: cluster 2023-11-17T07:03:21.092921+0000 mgr.smithi097.swvonl (mgr.14180) 642 : cluster [DBG] pgmap v486: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:23.862 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:23.862 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:23.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:22 smithi097 bash[24958]: cluster 2023-11-17T07:03:21.092921+0000 mgr.smithi097.swvonl (mgr.14180) 642 : cluster [DBG] pgmap v486: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:23.864 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:24.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:23 smithi097 bash[24958]: audit 2023-11-17T07:03:22.192003+0000 mgr.smithi097.swvonl (mgr.14180) 643 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:24.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:23 smithi193 bash[28559]: audit 2023-11-17T07:03:22.192003+0000 mgr.smithi097.swvonl (mgr.14180) 643 : audit [DBG] from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:24.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:24.874 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:24 smithi097 bash[24958]: cluster 2023-11-17T07:03:23.093969+0000 mgr.smithi097.swvonl (mgr.14180) 644 : cluster [DBG] pgmap v487: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:24 smithi193 bash[28559]: cluster 2023-11-17T07:03:23.093969+0000 mgr.smithi097.swvonl (mgr.14180) 644 : cluster [DBG] pgmap v487: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:27.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:26 smithi097 bash[24958]: cluster 2023-11-17T07:03:25.095396+0000 mgr.smithi097.swvonl (mgr.14180) 645 : cluster [DBG] pgmap v488: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:27.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:26 smithi193 bash[28559]: cluster 2023-11-17T07:03:25.095396+0000 mgr.smithi097.swvonl (mgr.14180) 645 : cluster [DBG] pgmap v488: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:28.405 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:28.405 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:28.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:28 smithi097 bash[24958]: cluster 2023-11-17T07:03:27.096430+0000 mgr.smithi097.swvonl (mgr.14180) 646 : cluster [DBG] pgmap v489: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:28 smithi193 bash[28559]: cluster 2023-11-17T07:03:27.096430+0000 mgr.smithi097.swvonl (mgr.14180) 646 : cluster [DBG] pgmap v489: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:29.190 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:30.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:29 smithi097 bash[24958]: audit 2023-11-17T07:03:28.395321+0000 mgr.smithi097.swvonl (mgr.14180) 647 : audit [DBG] from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:29 smithi193 bash[28559]: audit 2023-11-17T07:03:28.395321+0000 mgr.smithi097.swvonl (mgr.14180) 647 : audit [DBG] from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:30.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:30.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:30 smithi097 bash[24958]: cluster 2023-11-17T07:03:29.097542+0000 mgr.smithi097.swvonl (mgr.14180) 648 : cluster [DBG] pgmap v490: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:30 smithi193 bash[28559]: cluster 2023-11-17T07:03:29.097542+0000 mgr.smithi097.swvonl (mgr.14180) 648 : cluster [DBG] pgmap v490: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:32 smithi097 bash[24958]: cluster 2023-11-17T07:03:31.098503+0000 mgr.smithi097.swvonl (mgr.14180) 649 : cluster [DBG] pgmap v491: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:32 smithi193 bash[28559]: cluster 2023-11-17T07:03:31.098503+0000 mgr.smithi097.swvonl (mgr.14180) 649 : cluster [DBG] pgmap v491: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:33.938 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:33.939 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:34.759 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:34 smithi097 bash[24958]: cluster 2023-11-17T07:03:33.099186+0000 mgr.smithi097.swvonl (mgr.14180) 650 : cluster [DBG] pgmap v492: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:35.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:34 smithi193 bash[28559]: cluster 2023-11-17T07:03:33.099186+0000 mgr.smithi097.swvonl (mgr.14180) 650 : cluster [DBG] pgmap v492: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:35.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:35.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:35 smithi097 bash[24958]: audit 2023-11-17T07:03:33.924682+0000 mgr.smithi097.swvonl (mgr.14180) 651 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:36.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:35 smithi193 bash[28559]: audit 2023-11-17T07:03:33.924682+0000 mgr.smithi097.swvonl (mgr.14180) 651 : audit [DBG] from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:37.019 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:36 smithi097 bash[24958]: cluster 2023-11-17T07:03:35.100686+0000 mgr.smithi097.swvonl (mgr.14180) 652 : cluster [DBG] pgmap v493: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:36 smithi193 bash[28559]: cluster 2023-11-17T07:03:35.100686+0000 mgr.smithi097.swvonl (mgr.14180) 652 : cluster [DBG] pgmap v493: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:38.980 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:38 smithi097 bash[24958]: cluster 2023-11-17T07:03:37.101868+0000 mgr.smithi097.swvonl (mgr.14180) 653 : cluster [DBG] pgmap v494: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:39.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:38 smithi193 bash[28559]: cluster 2023-11-17T07:03:37.101868+0000 mgr.smithi097.swvonl (mgr.14180) 653 : cluster [DBG] pgmap v494: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:39.370 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:39.371 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:40.274 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:40 smithi097 bash[24958]: cluster 2023-11-17T07:03:39.102687+0000 mgr.smithi097.swvonl (mgr.14180) 654 : cluster [DBG] pgmap v495: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:40 smithi097 bash[24958]: audit 2023-11-17T07:03:39.362435+0000 mgr.smithi097.swvonl (mgr.14180) 655 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:40 smithi193 bash[28559]: cluster 2023-11-17T07:03:39.102687+0000 mgr.smithi097.swvonl (mgr.14180) 654 : cluster [DBG] pgmap v495: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:40 smithi193 bash[28559]: audit 2023-11-17T07:03:39.362435+0000 mgr.smithi097.swvonl (mgr.14180) 655 : audit [DBG] from='client.14768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:41.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:42.985 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:42 smithi097 bash[24958]: cluster 2023-11-17T07:03:41.103887+0000 mgr.smithi097.swvonl (mgr.14180) 656 : cluster [DBG] pgmap v496: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:55.739 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:44 smithi097 bash[24958]: cluster 2023-11-17T07:03:43.104895+0000 mgr.smithi097.swvonl (mgr.14180) 657 : cluster [DBG] pgmap v497: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.740 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:42 smithi193 bash[28559]: cluster 2023-11-17T07:03:41.103887+0000 mgr.smithi097.swvonl (mgr.14180) 656 : cluster [DBG] pgmap v496: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:44 smithi193 bash[28559]: cluster 2023-11-17T07:03:43.104895+0000 mgr.smithi097.swvonl (mgr.14180) 657 : cluster [DBG] pgmap v497: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:46 smithi193 bash[28559]: cluster 2023-11-17T07:03:45.105646+0000 mgr.smithi097.swvonl (mgr.14180) 658 : cluster [DBG] pgmap v498: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:46 smithi193 bash[28559]: audit 2023-11-17T07:03:45.309757+0000 mgr.smithi097.swvonl (mgr.14180) 659 : audit [DBG] from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:48 smithi193 bash[28559]: cluster 2023-11-17T07:03:47.106470+0000 mgr.smithi097.swvonl (mgr.14180) 660 : cluster [DBG] pgmap v499: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:50 smithi193 bash[28559]: cluster 2023-11-17T07:03:49.107554+0000 mgr.smithi097.swvonl (mgr.14180) 661 : cluster [DBG] pgmap v500: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:52 smithi193 bash[28559]: cluster 2023-11-17T07:03:51.108889+0000 mgr.smithi097.swvonl (mgr.14180) 662 : cluster [DBG] pgmap v501: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:54 smithi193 bash[28559]: cluster 2023-11-17T07:03:53.109854+0000 mgr.smithi097.swvonl (mgr.14180) 663 : cluster [DBG] pgmap v502: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.741 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:03:55.741 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:03:55.742 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:03:55.742 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:46 smithi097 bash[24958]: cluster 2023-11-17T07:03:45.105646+0000 mgr.smithi097.swvonl (mgr.14180) 658 : cluster [DBG] pgmap v498: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.742 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:46 smithi097 bash[24958]: audit 2023-11-17T07:03:45.309757+0000 mgr.smithi097.swvonl (mgr.14180) 659 : audit [DBG] from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:03:55.743 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:48 smithi097 bash[24958]: cluster 2023-11-17T07:03:47.106470+0000 mgr.smithi097.swvonl (mgr.14180) 660 : cluster [DBG] pgmap v499: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.743 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:50 smithi097 bash[24958]: cluster 2023-11-17T07:03:49.107554+0000 mgr.smithi097.swvonl (mgr.14180) 661 : cluster [DBG] pgmap v500: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:55.743 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:52 smithi097 bash[24958]: cluster 2023-11-17T07:03:51.108889+0000 mgr.smithi097.swvonl (mgr.14180) 662 : cluster [DBG] pgmap v501: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:03:55.743 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:54 smithi097 bash[24958]: cluster 2023-11-17T07:03:53.109854+0000 mgr.smithi097.swvonl (mgr.14180) 663 : cluster [DBG] pgmap v502: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:56.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:03:56.944 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:56 smithi097 bash[24958]: cluster 2023-11-17T07:03:55.110751+0000 mgr.smithi097.swvonl (mgr.14180) 664 : cluster [DBG] pgmap v503: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:56 smithi193 bash[28559]: cluster 2023-11-17T07:03:55.110751+0000 mgr.smithi097.swvonl (mgr.14180) 664 : cluster [DBG] pgmap v503: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:58.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:03:58 smithi097 bash[24958]: cluster 2023-11-17T07:03:57.111380+0000 mgr.smithi097.swvonl (mgr.14180) 665 : cluster [DBG] pgmap v504: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:03:59.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:03:58 smithi193 bash[28559]: cluster 2023-11-17T07:03:57.111380+0000 mgr.smithi097.swvonl (mgr.14180) 665 : cluster [DBG] pgmap v504: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:04:00.542 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:05:37.187 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:01:54.940151Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.jpgsta on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:02:59.022892Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.utofbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-utofbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.utofbr\nDeploy daemon haproxy.nfs.foo.smithi193.utofbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.074729Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pmshap on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pmshap\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pmshap\nDeploy daemon haproxy.nfs.foo.smithi097.pmshap ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:03:03.078962Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.tlrerg on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:03:03.082547Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fqiidp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:05:37.191 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:00 smithi097 bash[24958]: cluster 2023-11-17T07:03:59.112454+0000 mgr.smithi097.swvonl (mgr.14180) 666 : cluster [DBG] pgmap v505: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.194 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:00 smithi193 bash[28559]: cluster 2023-11-17T07:03:59.112454+0000 mgr.smithi097.swvonl (mgr.14180) 666 : cluster [DBG] pgmap v505: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:01 smithi193 bash[28559]: audit 2023-11-17T07:04:00.529327+0000 mgr.smithi097.swvonl (mgr.14180) 667 : audit [DBG] from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:02 smithi193 bash[28559]: cluster 2023-11-17T07:04:01.113730+0000 mgr.smithi097.swvonl (mgr.14180) 668 : cluster [DBG] pgmap v506: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.086171+0000 mon.smithi097 (mon.0) 833 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.396725+0000 mon.smithi097 (mon.0) 834 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:05:37.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.398047+0000 mon.smithi097 (mon.0) 835 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.404974+0000 mon.smithi097 (mon.0) 836 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.412958+0000 mon.smithi097 (mon.0) 837 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:03 smithi193 bash[28559]: audit 2023-11-17T07:04:03.429054+0000 mon.smithi097 (mon.0) 838 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:04 smithi193 bash[28559]: cluster 2023-11-17T07:04:03.114569+0000 mgr.smithi097.swvonl (mgr.14180) 669 : cluster [DBG] pgmap v507: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:04 smithi193 bash[28559]: cluster 2023-11-17T07:04:03.405892+0000 mgr.smithi097.swvonl (mgr.14180) 670 : cluster [DBG] pgmap v508: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:04 smithi193 bash[28559]: cephadm 2023-11-17T07:04:03.434310+0000 mgr.smithi097.swvonl (mgr.14180) 671 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.qotfyh on smithi193 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:04 smithi193 bash[28559]: cluster 2023-11-17T07:04:03.695690+0000 mon.smithi097 (mon.0) 839 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:04 smithi193 bash[28559]: cluster 2023-11-17T07:04:03.695771+0000 mon.smithi097 (mon.0) 840 : cluster [INF] Cluster is now healthy 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:06 smithi193 bash[28559]: cluster 2023-11-17T07:04:05.407043+0000 mgr.smithi097.swvonl (mgr.14180) 672 : cluster [DBG] pgmap v509: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:07 smithi193 bash[28559]: audit 2023-11-17T07:04:07.239702+0000 mon.smithi097 (mon.0) 841 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.qotfyh"}]: dispatch 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: cephadm 2023-11-17T07:04:07.238517+0000 mgr.smithi097.swvonl (mgr.14180) 673 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:05:37.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: cephadm 2023-11-17T07:04:07.239037+0000 mgr.smithi097.swvonl (mgr.14180) 674 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: cephadm 2023-11-17T07:04:07.241390+0000 mgr.smithi097.swvonl (mgr.14180) 675 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.qotfyh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: cephadm 2023-11-17T07:04:07.245998+0000 mgr.smithi097.swvonl (mgr.14180) 676 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.xqlbfi on smithi097 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:08 smithi193 bash[28559]: cluster 2023-11-17T07:04:07.407733+0000 mgr.smithi097.swvonl (mgr.14180) 677 : cluster [DBG] pgmap v510: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:10 smithi193 bash[28559]: cluster 2023-11-17T07:04:09.408545+0000 mgr.smithi097.swvonl (mgr.14180) 678 : cluster [DBG] pgmap v511: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:11 smithi193 bash[28559]: audit 2023-11-17T07:04:11.204961+0000 mon.smithi097 (mon.0) 842 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.xqlbfi"}]: dispatch 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:11 smithi193 bash[28559]: audit 2023-11-17T07:04:11.524008+0000 mon.smithi097 (mon.0) 843 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cephadm 2023-11-17T07:04:11.203671+0000 mgr.smithi097.swvonl (mgr.14180) 679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:05:37.258 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.259 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cephadm 2023-11-17T07:04:11.204205+0000 mgr.smithi097.swvonl (mgr.14180) 680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cephadm 2023-11-17T07:04:11.206364+0000 mgr.smithi097.swvonl (mgr.14180) 681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.xqlbfi on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.260 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cephadm 2023-11-17T07:04:11.208668+0000 mgr.smithi097.swvonl (mgr.14180) 682 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.brnwll on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cephadm 2023-11-17T07:04:11.210743+0000 mgr.smithi097.swvonl (mgr.14180) 683 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cluster 2023-11-17T07:04:11.212306+0000 mgr.smithi097.swvonl (mgr.14180) 684 : cluster [DBG] pgmap v512: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:12 smithi193 bash[28559]: cluster 2023-11-17T07:04:11.733314+0000 mon.smithi097 (mon.0) 844 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:14 smithi193 bash[28559]: cluster 2023-11-17T07:04:13.213366+0000 mgr.smithi097.swvonl (mgr.14180) 685 : cluster [DBG] pgmap v513: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:14 smithi193 bash[28559]: audit 2023-11-17T07:04:13.676584+0000 mon.smithi097 (mon.0) 845 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:16 smithi193 bash[28559]: cluster 2023-11-17T07:04:15.214831+0000 mgr.smithi097.swvonl (mgr.14180) 686 : cluster [DBG] pgmap v514: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:18 smithi193 bash[28559]: cluster 2023-11-17T07:04:17.215744+0000 mgr.smithi097.swvonl (mgr.14180) 687 : cluster [DBG] pgmap v515: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:20 smithi193 bash[28559]: cluster 2023-11-17T07:04:19.216834+0000 mgr.smithi097.swvonl (mgr.14180) 688 : cluster [DBG] pgmap v516: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.261 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:22 smithi193 bash[28559]: cluster 2023-11-17T07:04:21.218169+0000 mgr.smithi097.swvonl (mgr.14180) 689 : cluster [DBG] pgmap v517: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2023-11-17T07:05:37.262 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:24 smithi193 bash[28559]: cluster 2023-11-17T07:04:23.219223+0000 mgr.smithi097.swvonl (mgr.14180) 690 : cluster [DBG] pgmap v518: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.262 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:26 smithi193 bash[28559]: cluster 2023-11-17T07:04:25.220666+0000 mgr.smithi097.swvonl (mgr.14180) 691 : cluster [DBG] pgmap v519: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.262 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:26 smithi193 bash[28559]: audit 2023-11-17T07:04:26.524335+0000 mon.smithi097 (mon.0) 846 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.262 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:28 smithi193 bash[28559]: cluster 2023-11-17T07:04:27.221636+0000 mgr.smithi097.swvonl (mgr.14180) 692 : cluster [DBG] pgmap v520: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:30 smithi193 bash[28559]: cluster 2023-11-17T07:04:29.222664+0000 mgr.smithi097.swvonl (mgr.14180) 693 : cluster [DBG] pgmap v521: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:32 smithi193 bash[28559]: cluster 2023-11-17T07:04:31.223939+0000 mgr.smithi097.swvonl (mgr.14180) 694 : cluster [DBG] pgmap v522: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:34 smithi193 bash[28559]: cluster 2023-11-17T07:04:33.225058+0000 mgr.smithi097.swvonl (mgr.14180) 695 : cluster [DBG] pgmap v523: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:36 smithi193 bash[28559]: cluster 2023-11-17T07:04:35.226687+0000 mgr.smithi097.swvonl (mgr.14180) 696 : cluster [DBG] pgmap v524: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:38 smithi193 bash[28559]: cluster 2023-11-17T07:04:37.227593+0000 mgr.smithi097.swvonl (mgr.14180) 697 : cluster [DBG] pgmap v525: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:40 smithi193 bash[28559]: cluster 2023-11-17T07:04:39.228780+0000 mgr.smithi097.swvonl (mgr.14180) 698 : cluster [DBG] pgmap v526: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.297 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:42 smithi193 bash[28559]: cluster 2023-11-17T07:04:41.230288+0000 mgr.smithi097.swvonl (mgr.14180) 699 : cluster [DBG] pgmap v527: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:44 smithi193 bash[28559]: cluster 2023-11-17T07:04:43.231393+0000 mgr.smithi097.swvonl (mgr.14180) 700 : cluster [DBG] pgmap v528: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:46 smithi193 bash[28559]: cluster 2023-11-17T07:04:45.232836+0000 mgr.smithi097.swvonl (mgr.14180) 701 : cluster [DBG] pgmap v529: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:48 smithi193 bash[28559]: cluster 2023-11-17T07:04:47.233965+0000 mgr.smithi097.swvonl (mgr.14180) 702 : cluster [DBG] pgmap v530: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:50 smithi193 bash[28559]: cluster 2023-11-17T07:04:49.235229+0000 mgr.smithi097.swvonl (mgr.14180) 703 : cluster [DBG] pgmap v531: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:52 smithi193 bash[28559]: cluster 2023-11-17T07:04:51.236669+0000 mgr.smithi097.swvonl (mgr.14180) 704 : cluster [DBG] pgmap v532: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:54 smithi193 bash[28559]: cluster 2023-11-17T07:04:53.237632+0000 mgr.smithi097.swvonl (mgr.14180) 705 : cluster [DBG] pgmap v533: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:56 smithi193 bash[28559]: cluster 2023-11-17T07:04:55.239094+0000 mgr.smithi097.swvonl (mgr.14180) 706 : cluster [DBG] pgmap v534: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:04:58 smithi193 bash[28559]: cluster 2023-11-17T07:04:57.240061+0000 mgr.smithi097.swvonl (mgr.14180) 707 : cluster [DBG] pgmap v535: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:00 smithi193 bash[28559]: cluster 2023-11-17T07:04:59.241157+0000 mgr.smithi097.swvonl (mgr.14180) 708 : cluster [DBG] pgmap v536: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.298 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:02 smithi193 bash[28559]: cluster 2023-11-17T07:05:01.242382+0000 mgr.smithi097.swvonl (mgr.14180) 709 : cluster [DBG] pgmap v537: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:03 smithi193 bash[28559]: cluster 2023-11-17T07:05:03.243350+0000 mgr.smithi097.swvonl (mgr.14180) 710 : cluster [DBG] pgmap v538: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:06 smithi193 bash[28559]: cluster 2023-11-17T07:05:05.244774+0000 mgr.smithi097.swvonl (mgr.14180) 711 : cluster [DBG] pgmap v539: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:08 smithi193 bash[28559]: cluster 2023-11-17T07:05:07.245747+0000 mgr.smithi097.swvonl (mgr.14180) 712 : cluster [DBG] pgmap v540: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:10 smithi193 bash[28559]: cluster 2023-11-17T07:05:09.246750+0000 mgr.smithi097.swvonl (mgr.14180) 713 : cluster [DBG] pgmap v541: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:11 smithi193 bash[28559]: audit 2023-11-17T07:05:11.215136+0000 mon.smithi097 (mon.0) 847 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: cluster 2023-11-17T07:05:11.248035+0000 mgr.smithi097.swvonl (mgr.14180) 714 : cluster [DBG] pgmap v542: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: audit 2023-11-17T07:05:11.514773+0000 mon.smithi097 (mon.0) 848 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: audit 2023-11-17T07:05:11.516569+0000 mon.smithi097 (mon.0) 849 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: audit 2023-11-17T07:05:11.527229+0000 mon.smithi097 (mon.0) 850 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: cluster 2023-11-17T07:05:11.527995+0000 mgr.smithi097.swvonl (mgr.14180) 715 : cluster [DBG] pgmap v543: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: audit 2023-11-17T07:05:11.541979+0000 mon.smithi097 (mon.0) 851 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: audit 2023-11-17T07:05:11.556262+0000 mon.smithi097 (mon.0) 852 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:12 smithi193 bash[28559]: cephadm 2023-11-17T07:05:11.560225+0000 mgr.smithi097.swvonl (mgr.14180) 716 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jrktnx on smithi193 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:13 smithi193 bash[28559]: cluster 2023-11-17T07:05:12.524537+0000 mon.smithi097 (mon.0) 853 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:13 smithi193 bash[28559]: cluster 2023-11-17T07:05:12.524589+0000 mon.smithi097 (mon.0) 854 : cluster [INF] Cluster is now healthy 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:14 smithi193 bash[28559]: cluster 2023-11-17T07:05:13.528704+0000 mgr.smithi097.swvonl (mgr.14180) 717 : cluster [DBG] pgmap v544: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:15 smithi193 bash[28559]: audit 2023-11-17T07:05:15.346156+0000 mon.smithi097 (mon.0) 855 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jrktnx"}]: dispatch 2023-11-17T07:05:37.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: cephadm 2023-11-17T07:05:15.344896+0000 mgr.smithi097.swvonl (mgr.14180) 718 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:05:37.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: cephadm 2023-11-17T07:05:15.345440+0000 mgr.smithi097.swvonl (mgr.14180) 719 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: cephadm 2023-11-17T07:05:15.347976+0000 mgr.smithi097.swvonl (mgr.14180) 720 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: cephadm 2023-11-17T07:05:15.352810+0000 mgr.smithi097.swvonl (mgr.14180) 721 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.jhdovx on smithi097 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:16 smithi193 bash[28559]: cluster 2023-11-17T07:05:15.529835+0000 mgr.smithi097.swvonl (mgr.14180) 722 : cluster [DBG] pgmap v545: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:18 smithi193 bash[28559]: cluster 2023-11-17T07:05:17.530564+0000 mgr.smithi097.swvonl (mgr.14180) 723 : cluster [DBG] pgmap v546: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:19 smithi193 bash[28559]: audit 2023-11-17T07:05:19.123157+0000 mon.smithi097 (mon.0) 856 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.jhdovx"}]: dispatch 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cephadm 2023-11-17T07:05:19.121845+0000 mgr.smithi097.swvonl (mgr.14180) 724 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cephadm 2023-11-17T07:05:19.122390+0000 mgr.smithi097.swvonl (mgr.14180) 725 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cephadm 2023-11-17T07:05:19.124562+0000 mgr.smithi097.swvonl (mgr.14180) 726 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cephadm 2023-11-17T07:05:19.127755+0000 mgr.smithi097.swvonl (mgr.14180) 727 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cephadm 2023-11-17T07:05:19.130571+0000 mgr.smithi097.swvonl (mgr.14180) 728 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cluster 2023-11-17T07:05:19.131833+0000 mgr.smithi097.swvonl (mgr.14180) 729 : cluster [DBG] pgmap v547: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:20 smithi193 bash[28559]: cluster 2023-11-17T07:05:19.587360+0000 mon.smithi097 (mon.0) 857 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:05:37.335 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:22 smithi193 bash[28559]: cluster 2023-11-17T07:05:21.133189+0000 mgr.smithi097.swvonl (mgr.14180) 730 : cluster [DBG] pgmap v548: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:24 smithi193 bash[28559]: cluster 2023-11-17T07:05:23.134123+0000 mgr.smithi097.swvonl (mgr.14180) 731 : cluster [DBG] pgmap v549: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:24 smithi193 bash[28559]: audit 2023-11-17T07:05:23.731149+0000 mon.smithi097 (mon.0) 858 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:26 smithi193 bash[28559]: cluster 2023-11-17T07:05:25.135527+0000 mgr.smithi097.swvonl (mgr.14180) 732 : cluster [DBG] pgmap v550: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:28 smithi193 bash[28559]: cluster 2023-11-17T07:05:27.136546+0000 mgr.smithi097.swvonl (mgr.14180) 733 : cluster [DBG] pgmap v551: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:30 smithi193 bash[28559]: cluster 2023-11-17T07:05:29.137666+0000 mgr.smithi097.swvonl (mgr.14180) 734 : cluster [DBG] pgmap v552: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:32 smithi193 bash[28559]: cluster 2023-11-17T07:05:31.139027+0000 mgr.smithi097.swvonl (mgr.14180) 735 : cluster [DBG] pgmap v553: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:34 smithi193 bash[28559]: cluster 2023-11-17T07:05:33.140066+0000 mgr.smithi097.swvonl (mgr.14180) 736 : cluster [DBG] pgmap v554: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.336 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:36 smithi193 bash[28559]: cluster 2023-11-17T07:05:35.141376+0000 mgr.smithi097.swvonl (mgr.14180) 737 : cluster [DBG] pgmap v555: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:01 smithi097 bash[24958]: audit 2023-11-17T07:04:00.529327+0000 mgr.smithi097.swvonl (mgr.14180) 667 : audit [DBG] from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:02 smithi097 bash[24958]: cluster 2023-11-17T07:04:01.113730+0000 mgr.smithi097.swvonl (mgr.14180) 668 : cluster [DBG] pgmap v506: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.086171+0000 mon.smithi097 (mon.0) 833 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.396725+0000 mon.smithi097 (mon.0) 834 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.398047+0000 mon.smithi097 (mon.0) 835 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.404974+0000 mon.smithi097 (mon.0) 836 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.412958+0000 mon.smithi097 (mon.0) 837 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:05:37.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:03 smithi097 bash[24958]: audit 2023-11-17T07:04:03.429054+0000 mon.smithi097 (mon.0) 838 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:04 smithi097 bash[24958]: cluster 2023-11-17T07:04:03.114569+0000 mgr.smithi097.swvonl (mgr.14180) 669 : cluster [DBG] pgmap v507: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:04 smithi097 bash[24958]: cluster 2023-11-17T07:04:03.405892+0000 mgr.smithi097.swvonl (mgr.14180) 670 : cluster [DBG] pgmap v508: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:04 smithi097 bash[24958]: cephadm 2023-11-17T07:04:03.434310+0000 mgr.smithi097.swvonl (mgr.14180) 671 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.qotfyh on smithi193 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:04 smithi097 bash[24958]: cluster 2023-11-17T07:04:03.695690+0000 mon.smithi097 (mon.0) 839 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:04 smithi097 bash[24958]: cluster 2023-11-17T07:04:03.695771+0000 mon.smithi097 (mon.0) 840 : cluster [INF] Cluster is now healthy 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:06 smithi097 bash[24958]: cluster 2023-11-17T07:04:05.407043+0000 mgr.smithi097.swvonl (mgr.14180) 672 : cluster [DBG] pgmap v509: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:07 smithi097 bash[24958]: audit 2023-11-17T07:04:07.239702+0000 mon.smithi097 (mon.0) 841 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.qotfyh"}]: dispatch 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: cephadm 2023-11-17T07:04:07.238517+0000 mgr.smithi097.swvonl (mgr.14180) 673 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.340 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: cephadm 2023-11-17T07:04:07.239037+0000 mgr.smithi097.swvonl (mgr.14180) 674 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: cephadm 2023-11-17T07:04:07.241390+0000 mgr.smithi097.swvonl (mgr.14180) 675 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.qotfyh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qotfyh 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qotfyh 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qotfyh ... 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: cephadm 2023-11-17T07:04:07.245998+0000 mgr.smithi097.swvonl (mgr.14180) 676 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.xqlbfi on smithi097 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:08 smithi097 bash[24958]: cluster 2023-11-17T07:04:07.407733+0000 mgr.smithi097.swvonl (mgr.14180) 677 : cluster [DBG] pgmap v510: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:10 smithi097 bash[24958]: cluster 2023-11-17T07:04:09.408545+0000 mgr.smithi097.swvonl (mgr.14180) 678 : cluster [DBG] pgmap v511: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:11 smithi097 bash[24958]: audit 2023-11-17T07:04:11.204961+0000 mon.smithi097 (mon.0) 842 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.xqlbfi"}]: dispatch 2023-11-17T07:05:37.342 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:11 smithi097 bash[24958]: audit 2023-11-17T07:04:11.524008+0000 mon.smithi097 (mon.0) 843 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cephadm 2023-11-17T07:04:11.203671+0000 mgr.smithi097.swvonl (mgr.14180) 679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.360 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.361 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cephadm 2023-11-17T07:04:11.204205+0000 mgr.smithi097.swvonl (mgr.14180) 680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cephadm 2023-11-17T07:04:11.206364+0000 mgr.smithi097.swvonl (mgr.14180) 681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.xqlbfi on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-xqlbfi 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.362 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.xqlbfi 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.xqlbfi ... 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cephadm 2023-11-17T07:04:11.208668+0000 mgr.smithi097.swvonl (mgr.14180) 682 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.brnwll on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cephadm 2023-11-17T07:04:11.210743+0000 mgr.smithi097.swvonl (mgr.14180) 683 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cluster 2023-11-17T07:04:11.212306+0000 mgr.smithi097.swvonl (mgr.14180) 684 : cluster [DBG] pgmap v512: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:12 smithi097 bash[24958]: cluster 2023-11-17T07:04:11.733314+0000 mon.smithi097 (mon.0) 844 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:14 smithi097 bash[24958]: cluster 2023-11-17T07:04:13.213366+0000 mgr.smithi097.swvonl (mgr.14180) 685 : cluster [DBG] pgmap v513: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:05:37.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:14 smithi097 bash[24958]: audit 2023-11-17T07:04:13.676584+0000 mon.smithi097 (mon.0) 845 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:16 smithi097 bash[24958]: cluster 2023-11-17T07:04:15.214831+0000 mgr.smithi097.swvonl (mgr.14180) 686 : cluster [DBG] pgmap v514: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:18 smithi097 bash[24958]: cluster 2023-11-17T07:04:17.215744+0000 mgr.smithi097.swvonl (mgr.14180) 687 : cluster [DBG] pgmap v515: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:20 smithi097 bash[24958]: cluster 2023-11-17T07:04:19.216834+0000 mgr.smithi097.swvonl (mgr.14180) 688 : cluster [DBG] pgmap v516: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:22 smithi097 bash[24958]: cluster 2023-11-17T07:04:21.218169+0000 mgr.smithi097.swvonl (mgr.14180) 689 : cluster [DBG] pgmap v517: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 346 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:24 smithi097 bash[24958]: cluster 2023-11-17T07:04:23.219223+0000 mgr.smithi097.swvonl (mgr.14180) 690 : cluster [DBG] pgmap v518: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:26 smithi097 bash[24958]: cluster 2023-11-17T07:04:25.220666+0000 mgr.smithi097.swvonl (mgr.14180) 691 : cluster [DBG] pgmap v519: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:26 smithi097 bash[24958]: audit 2023-11-17T07:04:26.524335+0000 mon.smithi097 (mon.0) 846 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:28 smithi097 bash[24958]: cluster 2023-11-17T07:04:27.221636+0000 mgr.smithi097.swvonl (mgr.14180) 692 : cluster [DBG] pgmap v520: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:30 smithi097 bash[24958]: cluster 2023-11-17T07:04:29.222664+0000 mgr.smithi097.swvonl (mgr.14180) 693 : cluster [DBG] pgmap v521: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:32 smithi097 bash[24958]: cluster 2023-11-17T07:04:31.223939+0000 mgr.smithi097.swvonl (mgr.14180) 694 : cluster [DBG] pgmap v522: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:34 smithi097 bash[24958]: cluster 2023-11-17T07:04:33.225058+0000 mgr.smithi097.swvonl (mgr.14180) 695 : cluster [DBG] pgmap v523: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:36 smithi097 bash[24958]: cluster 2023-11-17T07:04:35.226687+0000 mgr.smithi097.swvonl (mgr.14180) 696 : cluster [DBG] pgmap v524: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:38 smithi097 bash[24958]: cluster 2023-11-17T07:04:37.227593+0000 mgr.smithi097.swvonl (mgr.14180) 697 : cluster [DBG] pgmap v525: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:40 smithi097 bash[24958]: cluster 2023-11-17T07:04:39.228780+0000 mgr.smithi097.swvonl (mgr.14180) 698 : cluster [DBG] pgmap v526: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:42 smithi097 bash[24958]: cluster 2023-11-17T07:04:41.230288+0000 mgr.smithi097.swvonl (mgr.14180) 699 : cluster [DBG] pgmap v527: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:44 smithi097 bash[24958]: cluster 2023-11-17T07:04:43.231393+0000 mgr.smithi097.swvonl (mgr.14180) 700 : cluster [DBG] pgmap v528: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:46 smithi097 bash[24958]: cluster 2023-11-17T07:04:45.232836+0000 mgr.smithi097.swvonl (mgr.14180) 701 : cluster [DBG] pgmap v529: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:48 smithi097 bash[24958]: cluster 2023-11-17T07:04:47.233965+0000 mgr.smithi097.swvonl (mgr.14180) 702 : cluster [DBG] pgmap v530: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:50 smithi097 bash[24958]: cluster 2023-11-17T07:04:49.235229+0000 mgr.smithi097.swvonl (mgr.14180) 703 : cluster [DBG] pgmap v531: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:52 smithi097 bash[24958]: cluster 2023-11-17T07:04:51.236669+0000 mgr.smithi097.swvonl (mgr.14180) 704 : cluster [DBG] pgmap v532: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:54 smithi097 bash[24958]: cluster 2023-11-17T07:04:53.237632+0000 mgr.smithi097.swvonl (mgr.14180) 705 : cluster [DBG] pgmap v533: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:56 smithi097 bash[24958]: cluster 2023-11-17T07:04:55.239094+0000 mgr.smithi097.swvonl (mgr.14180) 706 : cluster [DBG] pgmap v534: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:04:58 smithi097 bash[24958]: cluster 2023-11-17T07:04:57.240061+0000 mgr.smithi097.swvonl (mgr.14180) 707 : cluster [DBG] pgmap v535: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:00 smithi097 bash[24958]: cluster 2023-11-17T07:04:59.241157+0000 mgr.smithi097.swvonl (mgr.14180) 708 : cluster [DBG] pgmap v536: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:02 smithi097 bash[24958]: cluster 2023-11-17T07:05:01.242382+0000 mgr.smithi097.swvonl (mgr.14180) 709 : cluster [DBG] pgmap v537: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:03 smithi097 bash[24958]: cluster 2023-11-17T07:05:03.243350+0000 mgr.smithi097.swvonl (mgr.14180) 710 : cluster [DBG] pgmap v538: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:06 smithi097 bash[24958]: cluster 2023-11-17T07:05:05.244774+0000 mgr.smithi097.swvonl (mgr.14180) 711 : cluster [DBG] pgmap v539: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:08 smithi097 bash[24958]: cluster 2023-11-17T07:05:07.245747+0000 mgr.smithi097.swvonl (mgr.14180) 712 : cluster [DBG] pgmap v540: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:10 smithi097 bash[24958]: cluster 2023-11-17T07:05:09.246750+0000 mgr.smithi097.swvonl (mgr.14180) 713 : cluster [DBG] pgmap v541: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:11 smithi097 bash[24958]: audit 2023-11-17T07:05:11.215136+0000 mon.smithi097 (mon.0) 847 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: cluster 2023-11-17T07:05:11.248035+0000 mgr.smithi097.swvonl (mgr.14180) 714 : cluster [DBG] pgmap v542: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: audit 2023-11-17T07:05:11.514773+0000 mon.smithi097 (mon.0) 848 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: audit 2023-11-17T07:05:11.516569+0000 mon.smithi097 (mon.0) 849 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: audit 2023-11-17T07:05:11.527229+0000 mon.smithi097 (mon.0) 850 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: cluster 2023-11-17T07:05:11.527995+0000 mgr.smithi097.swvonl (mgr.14180) 715 : cluster [DBG] pgmap v543: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: audit 2023-11-17T07:05:11.541979+0000 mon.smithi097 (mon.0) 851 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: audit 2023-11-17T07:05:11.556262+0000 mon.smithi097 (mon.0) 852 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:12 smithi097 bash[24958]: cephadm 2023-11-17T07:05:11.560225+0000 mgr.smithi097.swvonl (mgr.14180) 716 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jrktnx on smithi193 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:13 smithi097 bash[24958]: cluster 2023-11-17T07:05:12.524537+0000 mon.smithi097 (mon.0) 853 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:13 smithi097 bash[24958]: cluster 2023-11-17T07:05:12.524589+0000 mon.smithi097 (mon.0) 854 : cluster [INF] Cluster is now healthy 2023-11-17T07:05:37.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:14 smithi097 bash[24958]: cluster 2023-11-17T07:05:13.528704+0000 mgr.smithi097.swvonl (mgr.14180) 717 : cluster [DBG] pgmap v544: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:15 smithi097 bash[24958]: audit 2023-11-17T07:05:15.346156+0000 mon.smithi097 (mon.0) 855 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jrktnx"}]: dispatch 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: cephadm 2023-11-17T07:05:15.344896+0000 mgr.smithi097.swvonl (mgr.14180) 718 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: cephadm 2023-11-17T07:05:15.345440+0000 mgr.smithi097.swvonl (mgr.14180) 719 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: cephadm 2023-11-17T07:05:15.347976+0000 mgr.smithi097.swvonl (mgr.14180) 720 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jrktnx ... 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: cephadm 2023-11-17T07:05:15.352810+0000 mgr.smithi097.swvonl (mgr.14180) 721 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.jhdovx on smithi097 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:16 smithi097 bash[24958]: cluster 2023-11-17T07:05:15.529835+0000 mgr.smithi097.swvonl (mgr.14180) 722 : cluster [DBG] pgmap v545: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:18 smithi097 bash[24958]: cluster 2023-11-17T07:05:17.530564+0000 mgr.smithi097.swvonl (mgr.14180) 723 : cluster [DBG] pgmap v546: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:19 smithi097 bash[24958]: audit 2023-11-17T07:05:19.123157+0000 mon.smithi097 (mon.0) 856 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.jhdovx"}]: dispatch 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cephadm 2023-11-17T07:05:19.121845+0000 mgr.smithi097.swvonl (mgr.14180) 724 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cephadm 2023-11-17T07:05:19.122390+0000 mgr.smithi097.swvonl (mgr.14180) 725 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cephadm 2023-11-17T07:05:19.124562+0000 mgr.smithi097.swvonl (mgr.14180) 726 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx 2023-11-17T07:05:37.372 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.jhdovx ... 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cephadm 2023-11-17T07:05:19.127755+0000 mgr.smithi097.swvonl (mgr.14180) 727 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cephadm 2023-11-17T07:05:19.130571+0000 mgr.smithi097.swvonl (mgr.14180) 728 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cluster 2023-11-17T07:05:19.131833+0000 mgr.smithi097.swvonl (mgr.14180) 729 : cluster [DBG] pgmap v547: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:20 smithi097 bash[24958]: cluster 2023-11-17T07:05:19.587360+0000 mon.smithi097 (mon.0) 857 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:22 smithi097 bash[24958]: cluster 2023-11-17T07:05:21.133189+0000 mgr.smithi097.swvonl (mgr.14180) 730 : cluster [DBG] pgmap v548: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:05:37.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:24 smithi097 bash[24958]: cluster 2023-11-17T07:05:23.134123+0000 mgr.smithi097.swvonl (mgr.14180) 731 : cluster [DBG] pgmap v549: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:24 smithi097 bash[24958]: audit 2023-11-17T07:05:23.731149+0000 mon.smithi097 (mon.0) 858 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:05:37.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:26 smithi097 bash[24958]: cluster 2023-11-17T07:05:25.135527+0000 mgr.smithi097.swvonl (mgr.14180) 732 : cluster [DBG] pgmap v550: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:28 smithi097 bash[24958]: cluster 2023-11-17T07:05:27.136546+0000 mgr.smithi097.swvonl (mgr.14180) 733 : cluster [DBG] pgmap v551: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.395 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:30 smithi097 bash[24958]: cluster 2023-11-17T07:05:29.137666+0000 mgr.smithi097.swvonl (mgr.14180) 734 : cluster [DBG] pgmap v552: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:05:37.395 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:32 smithi097 bash[24958]: cluster 2023-11-17T07:05:31.139027+0000 mgr.smithi097.swvonl (mgr.14180) 735 : cluster [DBG] pgmap v553: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:37.395 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:34 smithi097 bash[24958]: cluster 2023-11-17T07:05:33.140066+0000 mgr.smithi097.swvonl (mgr.14180) 736 : cluster [DBG] pgmap v554: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:37.396 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:36 smithi097 bash[24958]: cluster 2023-11-17T07:05:35.141376+0000 mgr.smithi097.swvonl (mgr.14180) 737 : cluster [DBG] pgmap v555: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:38.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:05:39.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:38 smithi097 bash[24958]: cluster 2023-11-17T07:05:37.142439+0000 mgr.smithi097.swvonl (mgr.14180) 738 : cluster [DBG] pgmap v556: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:39.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:38 smithi193 bash[28559]: cluster 2023-11-17T07:05:37.142439+0000 mgr.smithi097.swvonl (mgr.14180) 738 : cluster [DBG] pgmap v556: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:40.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:40 smithi097 bash[24958]: cluster 2023-11-17T07:05:39.143215+0000 mgr.smithi097.swvonl (mgr.14180) 739 : cluster [DBG] pgmap v557: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:41.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:40 smithi193 bash[28559]: cluster 2023-11-17T07:05:39.143215+0000 mgr.smithi097.swvonl (mgr.14180) 739 : cluster [DBG] pgmap v557: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:41.959 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:05:41.959 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:05:42.765 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:05:43.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:42 smithi097 bash[24958]: cluster 2023-11-17T07:05:41.144467+0000 mgr.smithi097.swvonl (mgr.14180) 740 : cluster [DBG] pgmap v558: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:43.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:42 smithi193 bash[28559]: cluster 2023-11-17T07:05:41.144467+0000 mgr.smithi097.swvonl (mgr.14180) 740 : cluster [DBG] pgmap v558: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:43.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:05:44.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:43 smithi097 bash[24958]: audit 2023-11-17T07:05:41.945246+0000 mgr.smithi097.swvonl (mgr.14180) 741 : audit [DBG] from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:44.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:43 smithi193 bash[28559]: audit 2023-11-17T07:05:41.945246+0000 mgr.smithi097.swvonl (mgr.14180) 741 : audit [DBG] from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:45.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:44 smithi097 bash[24958]: cluster 2023-11-17T07:05:43.145530+0000 mgr.smithi097.swvonl (mgr.14180) 742 : cluster [DBG] pgmap v559: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:45.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:44 smithi193 bash[28559]: cluster 2023-11-17T07:05:43.145530+0000 mgr.smithi097.swvonl (mgr.14180) 742 : cluster [DBG] pgmap v559: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:47.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:46 smithi193 bash[28559]: cluster 2023-11-17T07:05:45.146805+0000 mgr.smithi097.swvonl (mgr.14180) 743 : cluster [DBG] pgmap v560: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:47.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:46 smithi097 bash[24958]: cluster 2023-11-17T07:05:45.146805+0000 mgr.smithi097.swvonl (mgr.14180) 743 : cluster [DBG] pgmap v560: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:47.330 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:05:47.330 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:05:48.248 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:05:49.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:05:49.262 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:48 smithi097 bash[24958]: cluster 2023-11-17T07:05:47.147796+0000 mgr.smithi097.swvonl (mgr.14180) 744 : cluster [DBG] pgmap v561: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:49.262 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:48 smithi097 bash[24958]: audit 2023-11-17T07:05:47.315163+0000 mgr.smithi097.swvonl (mgr.14180) 745 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:48 smithi193 bash[28559]: cluster 2023-11-17T07:05:47.147796+0000 mgr.smithi097.swvonl (mgr.14180) 744 : cluster [DBG] pgmap v561: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:48 smithi193 bash[28559]: audit 2023-11-17T07:05:47.315163+0000 mgr.smithi097.swvonl (mgr.14180) 745 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:51.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:50 smithi097 bash[24958]: cluster 2023-11-17T07:05:49.149047+0000 mgr.smithi097.swvonl (mgr.14180) 746 : cluster [DBG] pgmap v562: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:51.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:50 smithi193 bash[28559]: cluster 2023-11-17T07:05:49.149047+0000 mgr.smithi097.swvonl (mgr.14180) 746 : cluster [DBG] pgmap v562: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:05:52.827 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:05:53.085 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:52 smithi097 bash[24958]: cluster 2023-11-17T07:05:51.150462+0000 mgr.smithi097.swvonl (mgr.14180) 747 : cluster [DBG] pgmap v563: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:53.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:52 smithi193 bash[28559]: cluster 2023-11-17T07:05:51.150462+0000 mgr.smithi097.swvonl (mgr.14180) 747 : cluster [DBG] pgmap v563: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:05:53.668 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:05:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:53 smithi097 bash[24958]: audit 2023-11-17T07:05:52.811698+0000 mgr.smithi097.swvonl (mgr.14180) 748 : audit [DBG] from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:54.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:53 smithi193 bash[28559]: audit 2023-11-17T07:05:52.811698+0000 mgr.smithi097.swvonl (mgr.14180) 748 : audit [DBG] from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:54.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:05:55.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:54 smithi097 bash[24958]: cluster 2023-11-17T07:05:53.151570+0000 mgr.smithi097.swvonl (mgr.14180) 749 : cluster [DBG] pgmap v564: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:55.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:54 smithi193 bash[28559]: cluster 2023-11-17T07:05:53.151570+0000 mgr.smithi097.swvonl (mgr.14180) 749 : cluster [DBG] pgmap v564: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:56.116 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:55 smithi097 bash[24958]: cluster 2023-11-17T07:05:55.152571+0000 mgr.smithi097.swvonl (mgr.14180) 750 : cluster [DBG] pgmap v565: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:56.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:55 smithi193 bash[28559]: cluster 2023-11-17T07:05:55.152571+0000 mgr.smithi097.swvonl (mgr.14180) 750 : cluster [DBG] pgmap v565: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:58.291 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:05:58.291 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:05:58.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:58 smithi097 bash[24958]: cluster 2023-11-17T07:05:57.153608+0000 mgr.smithi097.swvonl (mgr.14180) 751 : cluster [DBG] pgmap v566: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:58.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:58 smithi193 bash[28559]: cluster 2023-11-17T07:05:57.153608+0000 mgr.smithi097.swvonl (mgr.14180) 751 : cluster [DBG] pgmap v566: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:05:59.111 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:05:59.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:05:59 smithi097 bash[24958]: audit 2023-11-17T07:05:58.277584+0000 mgr.smithi097.swvonl (mgr.14180) 752 : audit [DBG] from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:05:59.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:05:59 smithi193 bash[28559]: audit 2023-11-17T07:05:58.277584+0000 mgr.smithi097.swvonl (mgr.14180) 752 : audit [DBG] from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:00.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:00.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:00 smithi097 bash[24958]: cluster 2023-11-17T07:05:59.154788+0000 mgr.smithi097.swvonl (mgr.14180) 753 : cluster [DBG] pgmap v567: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:00.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:00 smithi193 bash[28559]: cluster 2023-11-17T07:05:59.154788+0000 mgr.smithi097.swvonl (mgr.14180) 753 : cluster [DBG] pgmap v567: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:02.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:02 smithi097 bash[24958]: cluster 2023-11-17T07:06:01.156248+0000 mgr.smithi097.swvonl (mgr.14180) 754 : cluster [DBG] pgmap v568: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:02 smithi193 bash[28559]: cluster 2023-11-17T07:06:01.156248+0000 mgr.smithi097.swvonl (mgr.14180) 754 : cluster [DBG] pgmap v568: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:03.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:03.869 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:04.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:04 smithi097 bash[24958]: cluster 2023-11-17T07:06:03.156899+0000 mgr.smithi097.swvonl (mgr.14180) 755 : cluster [DBG] pgmap v569: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:04.676 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:04.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:04 smithi193 bash[28559]: cluster 2023-11-17T07:06:03.156899+0000 mgr.smithi097.swvonl (mgr.14180) 755 : cluster [DBG] pgmap v569: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:05.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:05 smithi193 bash[28559]: audit 2023-11-17T07:06:03.856125+0000 mgr.smithi097.swvonl (mgr.14180) 756 : audit [DBG] from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:05.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:05.691 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:05 smithi097 bash[24958]: audit 2023-11-17T07:06:03.856125+0000 mgr.smithi097.swvonl (mgr.14180) 756 : audit [DBG] from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:06.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:06 smithi097 bash[24958]: cluster 2023-11-17T07:06:05.158288+0000 mgr.smithi097.swvonl (mgr.14180) 757 : cluster [DBG] pgmap v570: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:06.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:06 smithi193 bash[28559]: cluster 2023-11-17T07:06:05.158288+0000 mgr.smithi097.swvonl (mgr.14180) 757 : cluster [DBG] pgmap v570: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:08.654 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:08 smithi097 bash[24958]: cluster 2023-11-17T07:06:07.159364+0000 mgr.smithi097.swvonl (mgr.14180) 758 : cluster [DBG] pgmap v571: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:08.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:08 smithi193 bash[28559]: cluster 2023-11-17T07:06:07.159364+0000 mgr.smithi097.swvonl (mgr.14180) 758 : cluster [DBG] pgmap v571: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:09.053 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:09.053 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:09.894 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:10 smithi097 bash[24958]: audit 2023-11-17T07:06:09.039660+0000 mgr.smithi097.swvonl (mgr.14180) 759 : audit [DBG] from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:10 smithi097 bash[24958]: cluster 2023-11-17T07:06:09.160410+0000 mgr.smithi097.swvonl (mgr.14180) 760 : cluster [DBG] pgmap v572: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:10 smithi193 bash[28559]: audit 2023-11-17T07:06:09.039660+0000 mgr.smithi097.swvonl (mgr.14180) 759 : audit [DBG] from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:10 smithi193 bash[28559]: cluster 2023-11-17T07:06:09.160410+0000 mgr.smithi097.swvonl (mgr.14180) 760 : cluster [DBG] pgmap v572: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:10.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:12.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:12 smithi097 bash[24958]: cluster 2023-11-17T07:06:11.161890+0000 mgr.smithi097.swvonl (mgr.14180) 761 : cluster [DBG] pgmap v573: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:12.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:12 smithi193 bash[28559]: cluster 2023-11-17T07:06:11.161890+0000 mgr.smithi097.swvonl (mgr.14180) 761 : cluster [DBG] pgmap v573: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:14.476 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:14.476 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:14.747 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:14 smithi097 bash[24958]: cluster 2023-11-17T07:06:13.163051+0000 mgr.smithi097.swvonl (mgr.14180) 762 : cluster [DBG] pgmap v574: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:14.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:14 smithi193 bash[28559]: cluster 2023-11-17T07:06:13.163051+0000 mgr.smithi097.swvonl (mgr.14180) 762 : cluster [DBG] pgmap v574: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:15.323 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:15.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:15 smithi193 bash[28559]: audit 2023-11-17T07:06:14.463427+0000 mgr.smithi097.swvonl (mgr.14180) 763 : audit [DBG] from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:15.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:15 smithi097 bash[24958]: audit 2023-11-17T07:06:14.463427+0000 mgr.smithi097.swvonl (mgr.14180) 763 : audit [DBG] from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:16.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:16.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:16 smithi097 bash[24958]: cluster 2023-11-17T07:06:15.163912+0000 mgr.smithi097.swvonl (mgr.14180) 764 : cluster [DBG] pgmap v575: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:16.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:16 smithi193 bash[28559]: cluster 2023-11-17T07:06:15.163912+0000 mgr.smithi097.swvonl (mgr.14180) 764 : cluster [DBG] pgmap v575: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:18 smithi097 bash[24958]: cluster 2023-11-17T07:06:17.164962+0000 mgr.smithi097.swvonl (mgr.14180) 765 : cluster [DBG] pgmap v576: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:18.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:18 smithi193 bash[28559]: cluster 2023-11-17T07:06:17.164962+0000 mgr.smithi097.swvonl (mgr.14180) 765 : cluster [DBG] pgmap v576: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:19.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:19 smithi097 bash[24958]: audit 2023-11-17T07:06:19.134912+0000 mon.smithi097 (mon.0) 859 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:06:19.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:19 smithi193 bash[28559]: audit 2023-11-17T07:06:19.134912+0000 mon.smithi097 (mon.0) 859 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:06:19.903 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:19.903 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:04:11.210620Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.oajsae on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:20.747 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: cluster 2023-11-17T07:06:19.166232+0000 mgr.smithi097.swvonl (mgr.14180) 766 : cluster [DBG] pgmap v577: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:20.747 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: audit 2023-11-17T07:06:19.438880+0000 mon.smithi097 (mon.0) 860 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: audit 2023-11-17T07:06:19.439891+0000 mon.smithi097 (mon.0) 861 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: audit 2023-11-17T07:06:19.446397+0000 mon.smithi097 (mon.0) 862 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: cluster 2023-11-17T07:06:19.447036+0000 mgr.smithi097.swvonl (mgr.14180) 767 : cluster [DBG] pgmap v578: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: audit 2023-11-17T07:06:19.451350+0000 mon.smithi097 (mon.0) 863 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: audit 2023-11-17T07:06:19.463555+0000 mon.smithi097 (mon.0) 864 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:20.748 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:20 smithi097 bash[24958]: cephadm 2023-11-17T07:06:19.476046+0000 mgr.smithi097.swvonl (mgr.14180) 768 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.guezdr on smithi193 2023-11-17T07:06:20.749 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:20.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: cluster 2023-11-17T07:06:19.166232+0000 mgr.smithi097.swvonl (mgr.14180) 766 : cluster [DBG] pgmap v577: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: audit 2023-11-17T07:06:19.438880+0000 mon.smithi097 (mon.0) 860 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: audit 2023-11-17T07:06:19.439891+0000 mon.smithi097 (mon.0) 861 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: audit 2023-11-17T07:06:19.446397+0000 mon.smithi097 (mon.0) 862 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: cluster 2023-11-17T07:06:19.447036+0000 mgr.smithi097.swvonl (mgr.14180) 767 : cluster [DBG] pgmap v578: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: audit 2023-11-17T07:06:19.451350+0000 mon.smithi097 (mon.0) 863 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: audit 2023-11-17T07:06:19.463555+0000 mon.smithi097 (mon.0) 864 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:20.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:20 smithi193 bash[28559]: cephadm 2023-11-17T07:06:19.476046+0000 mgr.smithi097.swvonl (mgr.14180) 768 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.guezdr on smithi193 2023-11-17T07:06:21.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:21.764 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:21 smithi097 bash[24958]: audit 2023-11-17T07:06:19.890571+0000 mgr.smithi097.swvonl (mgr.14180) 769 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:21.764 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:21 smithi097 bash[24958]: cluster 2023-11-17T07:06:20.447193+0000 mon.smithi097 (mon.0) 865 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:06:21.764 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:21 smithi097 bash[24958]: cluster 2023-11-17T07:06:20.447222+0000 mon.smithi097 (mon.0) 866 : cluster [INF] Cluster is now healthy 2023-11-17T07:06:21.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:21 smithi193 bash[28559]: audit 2023-11-17T07:06:19.890571+0000 mgr.smithi097.swvonl (mgr.14180) 769 : audit [DBG] from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:21.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:21 smithi193 bash[28559]: cluster 2023-11-17T07:06:20.447193+0000 mon.smithi097 (mon.0) 865 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:06:21.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:21 smithi193 bash[28559]: cluster 2023-11-17T07:06:20.447222+0000 mon.smithi097 (mon.0) 866 : cluster [INF] Cluster is now healthy 2023-11-17T07:06:22.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:22 smithi097 bash[24958]: cluster 2023-11-17T07:06:21.448027+0000 mgr.smithi097.swvonl (mgr.14180) 770 : cluster [DBG] pgmap v579: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:22.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:22 smithi193 bash[28559]: cluster 2023-11-17T07:06:21.448027+0000 mgr.smithi097.swvonl (mgr.14180) 770 : cluster [DBG] pgmap v579: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:23.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:23 smithi193 bash[28559]: audit 2023-11-17T07:06:23.359502+0000 mon.smithi097 (mon.0) 867 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.guezdr"}]: dispatch 2023-11-17T07:06:23.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:23 smithi097 bash[24958]: audit 2023-11-17T07:06:23.359502+0000 mon.smithi097 (mon.0) 867 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.guezdr"}]: dispatch 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: cephadm 2023-11-17T07:06:23.358318+0000 mgr.smithi097.swvonl (mgr.14180) 771 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: cephadm 2023-11-17T07:06:23.358906+0000 mgr.smithi097.swvonl (mgr.14180) 772 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: cephadm 2023-11-17T07:06:23.360895+0000 mgr.smithi097.swvonl (mgr.14180) 773 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: cephadm 2023-11-17T07:06:23.365502+0000 mgr.smithi097.swvonl (mgr.14180) 774 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.afwhex on smithi097 2023-11-17T07:06:24.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:24 smithi097 bash[24958]: cluster 2023-11-17T07:06:23.448484+0000 mgr.smithi097.swvonl (mgr.14180) 775 : cluster [DBG] pgmap v580: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: cephadm 2023-11-17T07:06:23.358318+0000 mgr.smithi097.swvonl (mgr.14180) 771 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:06:24.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: cephadm 2023-11-17T07:06:23.358906+0000 mgr.smithi097.swvonl (mgr.14180) 772 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: cephadm 2023-11-17T07:06:23.360895+0000 mgr.smithi097.swvonl (mgr.14180) 773 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.guezdr ... 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:24.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:24.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:24.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: cephadm 2023-11-17T07:06:23.365502+0000 mgr.smithi097.swvonl (mgr.14180) 774 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.afwhex on smithi097 2023-11-17T07:06:24.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:24 smithi193 bash[28559]: cluster 2023-11-17T07:06:23.448484+0000 mgr.smithi097.swvonl (mgr.14180) 775 : cluster [DBG] pgmap v580: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:25.361 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:25.361 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:15.347726Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jrktnx on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jrktnx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jrktnx\nDeploy daemon haproxy.nfs.foo.smithi193.jrktnx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.124400Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.jhdovx on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-jhdovx\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.jhdovx\nDeploy daemon haproxy.nfs.foo.smithi097.jhdovx ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:05:19.127604Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.lkgpxa on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:26.210 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:26.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:26 smithi097 bash[24958]: audit 2023-11-17T07:06:25.352727+0000 mgr.smithi097.swvonl (mgr.14180) 776 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:26.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:26 smithi097 bash[24958]: cluster 2023-11-17T07:06:25.449568+0000 mgr.smithi097.swvonl (mgr.14180) 777 : cluster [DBG] pgmap v581: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:26.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:26 smithi193 bash[28559]: audit 2023-11-17T07:06:25.352727+0000 mgr.smithi097.swvonl (mgr.14180) 776 : audit [DBG] from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:26.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:26 smithi193 bash[28559]: cluster 2023-11-17T07:06:25.449568+0000 mgr.smithi097.swvonl (mgr.14180) 777 : cluster [DBG] pgmap v581: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:06:27.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:27.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:27 smithi097 bash[24958]: audit 2023-11-17T07:06:26.528076+0000 mon.smithi097 (mon.0) 868 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:27.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:27 smithi097 bash[24958]: audit 2023-11-17T07:06:27.201966+0000 mon.smithi097 (mon.0) 869 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.afwhex"}]: dispatch 2023-11-17T07:06:27.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:27 smithi193 bash[28559]: audit 2023-11-17T07:06:26.528076+0000 mon.smithi097 (mon.0) 868 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:27.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:27 smithi193 bash[28559]: audit 2023-11-17T07:06:27.201966+0000 mon.smithi097 (mon.0) 869 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.afwhex"}]: dispatch 2023-11-17T07:06:28.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cephadm 2023-11-17T07:06:27.200705+0000 mgr.smithi097.swvonl (mgr.14180) 778 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:06:28.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cephadm 2023-11-17T07:06:27.201262+0000 mgr.smithi097.swvonl (mgr.14180) 779 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cephadm 2023-11-17T07:06:27.203506+0000 mgr.smithi097.swvonl (mgr.14180) 780 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cephadm 2023-11-17T07:06:27.207620+0000 mgr.smithi097.swvonl (mgr.14180) 781 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cephadm 2023-11-17T07:06:27.211025+0000 mgr.smithi097.swvonl (mgr.14180) 782 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cluster 2023-11-17T07:06:27.212009+0000 mgr.smithi097.swvonl (mgr.14180) 783 : cluster [DBG] pgmap v582: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:06:28.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:28 smithi097 bash[24958]: cluster 2023-11-17T07:06:27.524665+0000 mon.smithi097 (mon.0) 870 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:06:28.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cephadm 2023-11-17T07:06:27.200705+0000 mgr.smithi097.swvonl (mgr.14180) 778 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cephadm 2023-11-17T07:06:27.201262+0000 mgr.smithi097.swvonl (mgr.14180) 779 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cephadm 2023-11-17T07:06:27.203506+0000 mgr.smithi097.swvonl (mgr.14180) 780 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.afwhex ... 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:06:28.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cephadm 2023-11-17T07:06:27.207620+0000 mgr.smithi097.swvonl (mgr.14180) 781 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cephadm 2023-11-17T07:06:27.211025+0000 mgr.smithi097.swvonl (mgr.14180) 782 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cluster 2023-11-17T07:06:27.212009+0000 mgr.smithi097.swvonl (mgr.14180) 783 : cluster [DBG] pgmap v582: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:06:28.813 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:28 smithi193 bash[28559]: cluster 2023-11-17T07:06:27.524665+0000 mon.smithi097 (mon.0) 870 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:06:30.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:29 smithi097 bash[24958]: audit 2023-11-17T07:06:28.778458+0000 mon.smithi097 (mon.0) 871 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:30.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:29 smithi193 bash[28559]: audit 2023-11-17T07:06:28.778458+0000 mon.smithi097 (mon.0) 871 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:30.871 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:30.871 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:30 smithi097 bash[24958]: cluster 2023-11-17T07:06:29.213108+0000 mgr.smithi097.swvonl (mgr.14180) 784 : cluster [DBG] pgmap v583: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:06:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:30 smithi193 bash[28559]: cluster 2023-11-17T07:06:29.213108+0000 mgr.smithi097.swvonl (mgr.14180) 784 : cluster [DBG] pgmap v583: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:06:31.701 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:32.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:33.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:32 smithi097 bash[24958]: audit 2023-11-17T07:06:30.857830+0000 mgr.smithi097.swvonl (mgr.14180) 785 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:33.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:32 smithi097 bash[24958]: cluster 2023-11-17T07:06:31.214368+0000 mgr.smithi097.swvonl (mgr.14180) 786 : cluster [DBG] pgmap v584: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:06:33.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:32 smithi193 bash[28559]: audit 2023-11-17T07:06:30.857830+0000 mgr.smithi097.swvonl (mgr.14180) 785 : audit [DBG] from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:33.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:32 smithi193 bash[28559]: cluster 2023-11-17T07:06:31.214368+0000 mgr.smithi097.swvonl (mgr.14180) 786 : cluster [DBG] pgmap v584: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:06:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:34 smithi097 bash[24958]: cluster 2023-11-17T07:06:33.215346+0000 mgr.smithi097.swvonl (mgr.14180) 787 : cluster [DBG] pgmap v585: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:35.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:34 smithi193 bash[28559]: cluster 2023-11-17T07:06:33.215346+0000 mgr.smithi097.swvonl (mgr.14180) 787 : cluster [DBG] pgmap v585: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:36.399 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:36.400 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:36 smithi097 bash[24958]: cluster 2023-11-17T07:06:35.216373+0000 mgr.smithi097.swvonl (mgr.14180) 788 : cluster [DBG] pgmap v586: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:37.189 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:37.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:36 smithi193 bash[28559]: cluster 2023-11-17T07:06:35.216373+0000 mgr.smithi097.swvonl (mgr.14180) 788 : cluster [DBG] pgmap v586: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:38.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:38.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:37 smithi097 bash[24958]: audit 2023-11-17T07:06:36.385827+0000 mgr.smithi097.swvonl (mgr.14180) 789 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:38.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:37 smithi193 bash[28559]: audit 2023-11-17T07:06:36.385827+0000 mgr.smithi097.swvonl (mgr.14180) 789 : audit [DBG] from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:39.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:38 smithi097 bash[24958]: cluster 2023-11-17T07:06:37.217568+0000 mgr.smithi097.swvonl (mgr.14180) 790 : cluster [DBG] pgmap v587: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:39.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:38 smithi193 bash[28559]: cluster 2023-11-17T07:06:37.217568+0000 mgr.smithi097.swvonl (mgr.14180) 790 : cluster [DBG] pgmap v587: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:06:41.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:40 smithi097 bash[24958]: cluster 2023-11-17T07:06:39.218832+0000 mgr.smithi097.swvonl (mgr.14180) 791 : cluster [DBG] pgmap v588: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:41.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:40 smithi193 bash[28559]: cluster 2023-11-17T07:06:39.218832+0000 mgr.smithi097.swvonl (mgr.14180) 791 : cluster [DBG] pgmap v588: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:41.862 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:41.863 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:42.710 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:42.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:42 smithi097 bash[24958]: cluster 2023-11-17T07:06:41.220133+0000 mgr.smithi097.swvonl (mgr.14180) 792 : cluster [DBG] pgmap v589: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:42.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:42 smithi097 bash[24958]: audit 2023-11-17T07:06:41.528187+0000 mon.smithi097 (mon.0) 872 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:42 smithi193 bash[28559]: cluster 2023-11-17T07:06:41.220133+0000 mgr.smithi097.swvonl (mgr.14180) 792 : cluster [DBG] pgmap v589: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:42 smithi193 bash[28559]: audit 2023-11-17T07:06:41.528187+0000 mon.smithi097 (mon.0) 872 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:06:43.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:43.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:43 smithi097 bash[24958]: audit 2023-11-17T07:06:41.847795+0000 mgr.smithi097.swvonl (mgr.14180) 793 : audit [DBG] from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:43.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:43 smithi193 bash[28559]: audit 2023-11-17T07:06:41.847795+0000 mgr.smithi097.swvonl (mgr.14180) 793 : audit [DBG] from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:44 smithi097 bash[24958]: cluster 2023-11-17T07:06:43.221111+0000 mgr.smithi097.swvonl (mgr.14180) 794 : cluster [DBG] pgmap v590: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:44.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:44 smithi193 bash[28559]: cluster 2023-11-17T07:06:43.221111+0000 mgr.smithi097.swvonl (mgr.14180) 794 : cluster [DBG] pgmap v590: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:46.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:46 smithi097 bash[24958]: cluster 2023-11-17T07:06:45.222569+0000 mgr.smithi097.swvonl (mgr.14180) 795 : cluster [DBG] pgmap v591: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:46.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:46 smithi193 bash[28559]: cluster 2023-11-17T07:06:45.222569+0000 mgr.smithi097.swvonl (mgr.14180) 795 : cluster [DBG] pgmap v591: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:47.456 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:47.456 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:48.289 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:48.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:48 smithi193 bash[28559]: cluster 2023-11-17T07:06:47.223514+0000 mgr.smithi097.swvonl (mgr.14180) 796 : cluster [DBG] pgmap v592: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:48.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:48 smithi193 bash[28559]: audit 2023-11-17T07:06:47.443795+0000 mgr.smithi097.swvonl (mgr.14180) 797 : audit [DBG] from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:48 smithi097 bash[24958]: cluster 2023-11-17T07:06:47.223514+0000 mgr.smithi097.swvonl (mgr.14180) 796 : cluster [DBG] pgmap v592: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:49.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:48 smithi097 bash[24958]: audit 2023-11-17T07:06:47.443795+0000 mgr.smithi097.swvonl (mgr.14180) 797 : audit [DBG] from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:49.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:50.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:50 smithi097 bash[24958]: cluster 2023-11-17T07:06:49.224634+0000 mgr.smithi097.swvonl (mgr.14180) 798 : cluster [DBG] pgmap v593: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:51.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:50 smithi193 bash[28559]: cluster 2023-11-17T07:06:49.224634+0000 mgr.smithi097.swvonl (mgr.14180) 798 : cluster [DBG] pgmap v593: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:52.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:52 smithi097 bash[24958]: cluster 2023-11-17T07:06:51.225877+0000 mgr.smithi097.swvonl (mgr.14180) 799 : cluster [DBG] pgmap v594: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:52.899 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:06:52.899 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:06:53.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:52 smithi193 bash[28559]: cluster 2023-11-17T07:06:51.225877+0000 mgr.smithi097.swvonl (mgr.14180) 799 : cluster [DBG] pgmap v594: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:06:53.745 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:06:54.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:06:54.944 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:54 smithi097 bash[24958]: audit 2023-11-17T07:06:52.885730+0000 mgr.smithi097.swvonl (mgr.14180) 800 : audit [DBG] from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:54.944 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:54 smithi097 bash[24958]: cluster 2023-11-17T07:06:53.226836+0000 mgr.smithi097.swvonl (mgr.14180) 801 : cluster [DBG] pgmap v595: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:54 smithi193 bash[28559]: audit 2023-11-17T07:06:52.885730+0000 mgr.smithi097.swvonl (mgr.14180) 800 : audit [DBG] from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:06:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:54 smithi193 bash[28559]: cluster 2023-11-17T07:06:53.226836+0000 mgr.smithi097.swvonl (mgr.14180) 801 : cluster [DBG] pgmap v595: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:56 smithi097 bash[24958]: cluster 2023-11-17T07:06:55.228229+0000 mgr.smithi097.swvonl (mgr.14180) 802 : cluster [DBG] pgmap v596: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:06:57.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:56 smithi193 bash[28559]: cluster 2023-11-17T07:06:55.228229+0000 mgr.smithi097.swvonl (mgr.14180) 802 : cluster [DBG] pgmap v596: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:03.895 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:03.895 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:03.897 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:58 smithi097 bash[24958]: cluster 2023-11-17T07:06:57.229256+0000 mgr.smithi097.swvonl (mgr.14180) 803 : cluster [DBG] pgmap v597: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:03.898 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:58 smithi193 bash[28559]: cluster 2023-11-17T07:06:57.229256+0000 mgr.smithi097.swvonl (mgr.14180) 803 : cluster [DBG] pgmap v597: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:03.898 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:06:59 smithi193 bash[28559]: audit 2023-11-17T07:06:58.430734+0000 mgr.smithi097.swvonl (mgr.14180) 804 : audit [DBG] from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:03.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:00 smithi193 bash[28559]: cluster 2023-11-17T07:06:59.230497+0000 mgr.smithi097.swvonl (mgr.14180) 805 : cluster [DBG] pgmap v598: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:03.899 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:02 smithi193 bash[28559]: cluster 2023-11-17T07:07:01.231673+0000 mgr.smithi097.swvonl (mgr.14180) 806 : cluster [DBG] pgmap v599: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:03.901 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:03.901 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:06:59 smithi097 bash[24958]: audit 2023-11-17T07:06:58.430734+0000 mgr.smithi097.swvonl (mgr.14180) 804 : audit [DBG] from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:03.901 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:00 smithi097 bash[24958]: cluster 2023-11-17T07:06:59.230497+0000 mgr.smithi097.swvonl (mgr.14180) 805 : cluster [DBG] pgmap v598: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:03.901 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:02 smithi097 bash[24958]: cluster 2023-11-17T07:07:01.231673+0000 mgr.smithi097.swvonl (mgr.14180) 806 : cluster [DBG] pgmap v599: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:04.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:04.913 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:04 smithi097 bash[24958]: cluster 2023-11-17T07:07:03.232873+0000 mgr.smithi097.swvonl (mgr.14180) 807 : cluster [DBG] pgmap v600: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:04 smithi193 bash[28559]: cluster 2023-11-17T07:07:03.232873+0000 mgr.smithi097.swvonl (mgr.14180) 807 : cluster [DBG] pgmap v600: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:07.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:06 smithi097 bash[24958]: cluster 2023-11-17T07:07:05.234454+0000 mgr.smithi097.swvonl (mgr.14180) 808 : cluster [DBG] pgmap v601: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:07.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:06 smithi193 bash[28559]: cluster 2023-11-17T07:07:05.234454+0000 mgr.smithi097.swvonl (mgr.14180) 808 : cluster [DBG] pgmap v601: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:08.592 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:13.086 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:13.087 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:08 smithi097 bash[24958]: cluster 2023-11-17T07:07:07.235451+0000 mgr.smithi097.swvonl (mgr.14180) 809 : cluster [DBG] pgmap v602: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:13.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:08 smithi193 bash[28559]: cluster 2023-11-17T07:07:07.235451+0000 mgr.smithi097.swvonl (mgr.14180) 809 : cluster [DBG] pgmap v602: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:13.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:09 smithi193 bash[28559]: audit 2023-11-17T07:07:08.581182+0000 mgr.smithi097.swvonl (mgr.14180) 810 : audit [DBG] from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:13.087 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:10 smithi193 bash[28559]: cluster 2023-11-17T07:07:09.236589+0000 mgr.smithi097.swvonl (mgr.14180) 811 : cluster [DBG] pgmap v603: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:13.088 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:12 smithi193 bash[28559]: cluster 2023-11-17T07:07:11.238008+0000 mgr.smithi097.swvonl (mgr.14180) 812 : cluster [DBG] pgmap v604: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:13.088 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:13.089 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:09 smithi097 bash[24958]: audit 2023-11-17T07:07:08.581182+0000 mgr.smithi097.swvonl (mgr.14180) 810 : audit [DBG] from='client.14840 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:13.089 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:10 smithi097 bash[24958]: cluster 2023-11-17T07:07:09.236589+0000 mgr.smithi097.swvonl (mgr.14180) 811 : cluster [DBG] pgmap v603: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:13.089 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:12 smithi097 bash[24958]: cluster 2023-11-17T07:07:11.238008+0000 mgr.smithi097.swvonl (mgr.14180) 812 : cluster [DBG] pgmap v604: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:14.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:15.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:14 smithi097 bash[24958]: cluster 2023-11-17T07:07:13.239086+0000 mgr.smithi097.swvonl (mgr.14180) 813 : cluster [DBG] pgmap v605: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:14 smithi193 bash[28559]: cluster 2023-11-17T07:07:13.239086+0000 mgr.smithi097.swvonl (mgr.14180) 813 : cluster [DBG] pgmap v605: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:17.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:16 smithi097 bash[24958]: cluster 2023-11-17T07:07:15.240139+0000 mgr.smithi097.swvonl (mgr.14180) 814 : cluster [DBG] pgmap v606: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:17.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:16 smithi193 bash[28559]: cluster 2023-11-17T07:07:15.240139+0000 mgr.smithi097.swvonl (mgr.14180) 814 : cluster [DBG] pgmap v606: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:17.604 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:17.604 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:18.486 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:18.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:18 smithi097 bash[24958]: cluster 2023-11-17T07:07:17.241008+0000 mgr.smithi097.swvonl (mgr.14180) 815 : cluster [DBG] pgmap v607: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:18.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:18 smithi097 bash[24958]: audit 2023-11-17T07:07:17.595542+0000 mgr.smithi097.swvonl (mgr.14180) 816 : audit [DBG] from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:18 smithi193 bash[28559]: cluster 2023-11-17T07:07:17.241008+0000 mgr.smithi097.swvonl (mgr.14180) 815 : cluster [DBG] pgmap v607: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:18 smithi193 bash[28559]: audit 2023-11-17T07:07:17.595542+0000 mgr.smithi097.swvonl (mgr.14180) 816 : audit [DBG] from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:19.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:20.940 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:20 smithi097 bash[24958]: cluster 2023-11-17T07:07:19.242173+0000 mgr.smithi097.swvonl (mgr.14180) 817 : cluster [DBG] pgmap v608: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:20 smithi193 bash[28559]: cluster 2023-11-17T07:07:19.242173+0000 mgr.smithi097.swvonl (mgr.14180) 817 : cluster [DBG] pgmap v608: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:22.946 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:22 smithi097 bash[24958]: cluster 2023-11-17T07:07:21.243463+0000 mgr.smithi097.swvonl (mgr.14180) 818 : cluster [DBG] pgmap v609: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:23.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:22 smithi193 bash[28559]: cluster 2023-11-17T07:07:21.243463+0000 mgr.smithi097.swvonl (mgr.14180) 818 : cluster [DBG] pgmap v609: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:07:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:23.318 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:24.152 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:24 smithi097 bash[24958]: cluster 2023-11-17T07:07:23.244596+0000 mgr.smithi097.swvonl (mgr.14180) 819 : cluster [DBG] pgmap v610: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:24 smithi097 bash[24958]: audit 2023-11-17T07:07:23.306665+0000 mgr.smithi097.swvonl (mgr.14180) 820 : audit [DBG] from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:25.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:24 smithi193 bash[28559]: cluster 2023-11-17T07:07:23.244596+0000 mgr.smithi097.swvonl (mgr.14180) 819 : cluster [DBG] pgmap v610: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:25.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:24 smithi193 bash[28559]: audit 2023-11-17T07:07:23.306665+0000 mgr.smithi097.swvonl (mgr.14180) 820 : audit [DBG] from='client.14848 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:25.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:27.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:26 smithi097 bash[24958]: cluster 2023-11-17T07:07:25.245940+0000 mgr.smithi097.swvonl (mgr.14180) 821 : cluster [DBG] pgmap v611: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:27.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:26 smithi193 bash[28559]: cluster 2023-11-17T07:07:25.245940+0000 mgr.smithi097.swvonl (mgr.14180) 821 : cluster [DBG] pgmap v611: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:28.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:27 smithi097 bash[24958]: audit 2023-11-17T07:07:27.215215+0000 mon.smithi097 (mon.0) 873 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:07:28.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:27 smithi097 bash[24958]: audit 2023-11-17T07:07:27.483537+0000 mon.smithi097 (mon.0) 874 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:27 smithi193 bash[28559]: audit 2023-11-17T07:07:27.215215+0000 mon.smithi097 (mon.0) 873 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:07:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:27 smithi193 bash[28559]: audit 2023-11-17T07:07:27.483537+0000 mon.smithi097 (mon.0) 874 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:28.701 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:28.701 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:05:19.130418Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pqrmlr on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:28.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: cluster 2023-11-17T07:07:27.246534+0000 mgr.smithi097.swvonl (mgr.14180) 822 : cluster [DBG] pgmap v612: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:28.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: audit 2023-11-17T07:07:27.748993+0000 mon.smithi097 (mon.0) 875 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:07:28.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: audit 2023-11-17T07:07:27.750376+0000 mon.smithi097 (mon.0) 876 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:07:28.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: audit 2023-11-17T07:07:27.756918+0000 mon.smithi097 (mon.0) 877 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:28.956 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: cluster 2023-11-17T07:07:27.757611+0000 mgr.smithi097.swvonl (mgr.14180) 823 : cluster [DBG] pgmap v613: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:28.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: audit 2023-11-17T07:07:27.761833+0000 mon.smithi097 (mon.0) 878 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:07:28.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: audit 2023-11-17T07:07:27.774298+0000 mon.smithi097 (mon.0) 879 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:28.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:28 smithi097 bash[24958]: cephadm 2023-11-17T07:07:27.779522+0000 mgr.smithi097.swvonl (mgr.14180) 824 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.puzwzl on smithi193 2023-11-17T07:07:29.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: cluster 2023-11-17T07:07:27.246534+0000 mgr.smithi097.swvonl (mgr.14180) 822 : cluster [DBG] pgmap v612: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: audit 2023-11-17T07:07:27.748993+0000 mon.smithi097 (mon.0) 875 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: audit 2023-11-17T07:07:27.750376+0000 mon.smithi097 (mon.0) 876 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: audit 2023-11-17T07:07:27.756918+0000 mon.smithi097 (mon.0) 877 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: cluster 2023-11-17T07:07:27.757611+0000 mgr.smithi097.swvonl (mgr.14180) 823 : cluster [DBG] pgmap v613: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: audit 2023-11-17T07:07:27.761833+0000 mon.smithi097 (mon.0) 878 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: audit 2023-11-17T07:07:27.774298+0000 mon.smithi097 (mon.0) 879 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:28 smithi193 bash[28559]: cephadm 2023-11-17T07:07:27.779522+0000 mgr.smithi097.swvonl (mgr.14180) 824 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.puzwzl on smithi193 2023-11-17T07:07:29.515 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:29.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:29 smithi097 bash[24958]: audit 2023-11-17T07:07:28.690713+0000 mgr.smithi097.swvonl (mgr.14180) 825 : audit [DBG] from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:29.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:29 smithi097 bash[24958]: cluster 2023-11-17T07:07:28.755142+0000 mon.smithi097 (mon.0) 880 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:07:29.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:29 smithi097 bash[24958]: cluster 2023-11-17T07:07:28.755230+0000 mon.smithi097 (mon.0) 881 : cluster [INF] Cluster is now healthy 2023-11-17T07:07:30.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:29 smithi193 bash[28559]: audit 2023-11-17T07:07:28.690713+0000 mgr.smithi097.swvonl (mgr.14180) 825 : audit [DBG] from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:29 smithi193 bash[28559]: cluster 2023-11-17T07:07:28.755142+0000 mon.smithi097 (mon.0) 880 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:07:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:29 smithi193 bash[28559]: cluster 2023-11-17T07:07:28.755230+0000 mon.smithi097 (mon.0) 881 : cluster [INF] Cluster is now healthy 2023-11-17T07:07:30.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:30.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:30 smithi097 bash[24958]: cluster 2023-11-17T07:07:29.758626+0000 mgr.smithi097.swvonl (mgr.14180) 826 : cluster [DBG] pgmap v614: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 389 B/s rd, 0 op/s 2023-11-17T07:07:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:30 smithi193 bash[28559]: cluster 2023-11-17T07:07:29.758626+0000 mgr.smithi097.swvonl (mgr.14180) 826 : cluster [DBG] pgmap v614: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 389 B/s rd, 0 op/s 2023-11-17T07:07:32.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: cephadm 2023-11-17T07:07:31.712331+0000 mgr.smithi097.swvonl (mgr.14180) 827 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:07:32.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: cephadm 2023-11-17T07:07:31.712771+0000 mgr.smithi097.swvonl (mgr.14180) 828 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: audit 2023-11-17T07:07:31.713472+0000 mon.smithi097 (mon.0) 882 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.puzwzl"}]: dispatch 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: cephadm 2023-11-17T07:07:31.715368+0000 mgr.smithi097.swvonl (mgr.14180) 829 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: cephadm 2023-11-17T07:07:31.720059+0000 mgr.smithi097.swvonl (mgr.14180) 830 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.duxbgg on smithi097 2023-11-17T07:07:32.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:32 smithi097 bash[24958]: cluster 2023-11-17T07:07:31.759326+0000 mgr.smithi097.swvonl (mgr.14180) 831 : cluster [DBG] pgmap v615: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: cephadm 2023-11-17T07:07:31.712331+0000 mgr.smithi097.swvonl (mgr.14180) 827 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:07:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: cephadm 2023-11-17T07:07:31.712771+0000 mgr.smithi097.swvonl (mgr.14180) 828 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: audit 2023-11-17T07:07:31.713472+0000 mon.smithi097 (mon.0) 882 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.puzwzl"}]: dispatch 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: cephadm 2023-11-17T07:07:31.715368+0000 mgr.smithi097.swvonl (mgr.14180) 829 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.puzwzl ... 2023-11-17T07:07:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: cephadm 2023-11-17T07:07:31.720059+0000 mgr.smithi097.swvonl (mgr.14180) 830 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.duxbgg on smithi097 2023-11-17T07:07:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:32 smithi193 bash[28559]: cluster 2023-11-17T07:07:31.759326+0000 mgr.smithi097.swvonl (mgr.14180) 831 : cluster [DBG] pgmap v615: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:34.222 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:34.222 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:06:23.360685Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.guezdr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-guezdr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.guezdr\nDeploy daemon haproxy.nfs.foo.smithi193.guezdr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.203195Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.afwhex on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-afwhex\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.afwhex\nDeploy daemon haproxy.nfs.foo.smithi097.afwhex ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:06:27.207417Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xuoaoh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:31.715083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\nDeploy daemon haproxy.nfs.foo.smithi193.puzwzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:35.032 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:35.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:34 smithi097 bash[24958]: cluster 2023-11-17T07:07:33.760196+0000 mgr.smithi097.swvonl (mgr.14180) 832 : cluster [DBG] pgmap v616: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:34 smithi193 bash[28559]: cluster 2023-11-17T07:07:33.760196+0000 mgr.smithi097.swvonl (mgr.14180) 832 : cluster [DBG] pgmap v616: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:07:36.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:36.228 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:35 smithi097 bash[24958]: audit 2023-11-17T07:07:34.208706+0000 mgr.smithi097.swvonl (mgr.14180) 833 : audit [DBG] from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:36.229 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:35 smithi097 bash[24958]: audit 2023-11-17T07:07:35.607307+0000 mon.smithi097 (mon.0) 883 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.duxbgg"}]: dispatch 2023-11-17T07:07:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:35 smithi193 bash[28559]: audit 2023-11-17T07:07:34.208706+0000 mgr.smithi097.swvonl (mgr.14180) 833 : audit [DBG] from='client.14856 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:35 smithi193 bash[28559]: audit 2023-11-17T07:07:35.607307+0000 mon.smithi097 (mon.0) 883 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.duxbgg"}]: dispatch 2023-11-17T07:07:37.215 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cephadm 2023-11-17T07:07:35.606493+0000 mgr.smithi097.swvonl (mgr.14180) 834 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.215 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.215 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.215 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.215 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.216 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cephadm 2023-11-17T07:07:35.606753+0000 mgr.smithi097.swvonl (mgr.14180) 835 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cephadm 2023-11-17T07:07:35.608627+0000 mgr.smithi097.swvonl (mgr.14180) 836 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cephadm 2023-11-17T07:07:35.612602+0000 mgr.smithi097.swvonl (mgr.14180) 837 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:07:37.218 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cephadm 2023-11-17T07:07:35.615465+0000 mgr.smithi097.swvonl (mgr.14180) 838 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:07:37.219 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cluster 2023-11-17T07:07:35.616825+0000 mgr.smithi097.swvonl (mgr.14180) 839 : cluster [DBG] pgmap v617: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:07:37.219 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:36 smithi097 bash[24958]: cluster 2023-11-17T07:07:35.817507+0000 mon.smithi097 (mon.0) 884 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:07:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cephadm 2023-11-17T07:07:35.606493+0000 mgr.smithi097.swvonl (mgr.14180) 834 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cephadm 2023-11-17T07:07:35.606753+0000 mgr.smithi097.swvonl (mgr.14180) 835 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cephadm 2023-11-17T07:07:35.608627+0000 mgr.smithi097.swvonl (mgr.14180) 836 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.duxbgg ... 2023-11-17T07:07:37.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cephadm 2023-11-17T07:07:35.612602+0000 mgr.smithi097.swvonl (mgr.14180) 837 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:07:37.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cephadm 2023-11-17T07:07:35.615465+0000 mgr.smithi097.swvonl (mgr.14180) 838 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:07:37.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cluster 2023-11-17T07:07:35.616825+0000 mgr.smithi097.swvonl (mgr.14180) 839 : cluster [DBG] pgmap v617: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:07:37.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:36 smithi193 bash[28559]: cluster 2023-11-17T07:07:35.817507+0000 mon.smithi097 (mon.0) 884 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:07:39.159 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:38 smithi097 bash[24958]: cluster 2023-11-17T07:07:37.617850+0000 mgr.smithi097.swvonl (mgr.14180) 840 : cluster [DBG] pgmap v618: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:07:39.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:38 smithi193 bash[28559]: cluster 2023-11-17T07:07:37.617850+0000 mgr.smithi097.swvonl (mgr.14180) 840 : cluster [DBG] pgmap v618: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:07:39.511 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:39.511 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:31.715083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\nDeploy daemon haproxy.nfs.foo.smithi193.puzwzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.608500Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\nDeploy daemon haproxy.nfs.foo.smithi097.duxbgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.612429Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:40.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:39 smithi097 bash[24958]: audit 2023-11-17T07:07:38.830561+0000 mon.smithi097 (mon.0) 885 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:39 smithi193 bash[28559]: audit 2023-11-17T07:07:38.830561+0000 mon.smithi097 (mon.0) 885 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:07:40.367 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:41.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:40 smithi097 bash[24958]: audit 2023-11-17T07:07:39.502472+0000 mgr.smithi097.swvonl (mgr.14180) 841 : audit [DBG] from='client.14860 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:41.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:40 smithi097 bash[24958]: cluster 2023-11-17T07:07:39.619043+0000 mgr.smithi097.swvonl (mgr.14180) 842 : cluster [DBG] pgmap v619: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:41.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:40 smithi193 bash[28559]: audit 2023-11-17T07:07:39.502472+0000 mgr.smithi097.swvonl (mgr.14180) 841 : audit [DBG] from='client.14860 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:41.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:40 smithi193 bash[28559]: cluster 2023-11-17T07:07:39.619043+0000 mgr.smithi097.swvonl (mgr.14180) 842 : cluster [DBG] pgmap v619: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:41.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:42.678 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:42 smithi097 bash[24958]: cluster 2023-11-17T07:07:41.619671+0000 mgr.smithi097.swvonl (mgr.14180) 843 : cluster [DBG] pgmap v620: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:42 smithi193 bash[28559]: cluster 2023-11-17T07:07:41.619671+0000 mgr.smithi097.swvonl (mgr.14180) 843 : cluster [DBG] pgmap v620: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:44 smithi097 bash[24958]: cluster 2023-11-17T07:07:43.620794+0000 mgr.smithi097.swvonl (mgr.14180) 844 : cluster [DBG] pgmap v621: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:44.948 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:44.948 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:31.715083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\nDeploy daemon haproxy.nfs.foo.smithi193.puzwzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.608500Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\nDeploy daemon haproxy.nfs.foo.smithi097.duxbgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.612429Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:45.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:44 smithi193 bash[28559]: cluster 2023-11-17T07:07:43.620794+0000 mgr.smithi097.swvonl (mgr.14180) 844 : cluster [DBG] pgmap v621: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:45.787 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:46.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:46.990 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:46 smithi097 bash[24958]: audit 2023-11-17T07:07:44.935877+0000 mgr.smithi097.swvonl (mgr.14180) 845 : audit [DBG] from='client.14864 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:46.990 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:46 smithi097 bash[24958]: cluster 2023-11-17T07:07:45.621581+0000 mgr.smithi097.swvonl (mgr.14180) 846 : cluster [DBG] pgmap v622: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:47.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:46 smithi193 bash[28559]: audit 2023-11-17T07:07:44.935877+0000 mgr.smithi097.swvonl (mgr.14180) 845 : audit [DBG] from='client.14864 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:47.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:46 smithi193 bash[28559]: cluster 2023-11-17T07:07:45.621581+0000 mgr.smithi097.swvonl (mgr.14180) 846 : cluster [DBG] pgmap v622: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2023-11-17T07:07:49.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:48 smithi097 bash[24958]: cluster 2023-11-17T07:07:47.622686+0000 mgr.smithi097.swvonl (mgr.14180) 847 : cluster [DBG] pgmap v623: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:49.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:48 smithi193 bash[28559]: cluster 2023-11-17T07:07:47.622686+0000 mgr.smithi097.swvonl (mgr.14180) 847 : cluster [DBG] pgmap v623: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:50.277 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:07:50.277 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:31.715083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\nDeploy daemon haproxy.nfs.foo.smithi193.puzwzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.608500Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\nDeploy daemon haproxy.nfs.foo.smithi097.duxbgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.612429Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:07:50.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:50 smithi097 bash[24958]: cluster 2023-11-17T07:07:49.623769+0000 mgr.smithi097.swvonl (mgr.14180) 848 : cluster [DBG] pgmap v624: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:51.037 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:07:51.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:50 smithi193 bash[28559]: cluster 2023-11-17T07:07:49.623769+0000 mgr.smithi097.swvonl (mgr.14180) 848 : cluster [DBG] pgmap v624: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:07:52.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:07:52.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:51 smithi097 bash[24958]: audit 2023-11-17T07:07:50.268165+0000 mgr.smithi097.swvonl (mgr.14180) 849 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:52.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:51 smithi193 bash[28559]: audit 2023-11-17T07:07:50.268165+0000 mgr.smithi097.swvonl (mgr.14180) 849 : audit [DBG] from='client.14868 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:07:53.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:52 smithi097 bash[24958]: cluster 2023-11-17T07:07:51.625122+0000 mgr.smithi097.swvonl (mgr.14180) 850 : cluster [DBG] pgmap v625: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:54 smithi097 bash[24958]: cluster 2023-11-17T07:07:53.626273+0000 mgr.smithi097.swvonl (mgr.14180) 851 : cluster [DBG] pgmap v626: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.410 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:09:18.410 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T06:58:22.798086Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T06:58:22.352331Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T06:58:22.798358Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:06:27.210906Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.copjiu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:31.715083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.puzwzl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-puzwzl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.puzwzl\nDeploy daemon haproxy.nfs.foo.smithi193.puzwzl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.608500Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.duxbgg on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-duxbgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.duxbgg\nDeploy daemon haproxy.nfs.foo.smithi097.duxbgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:07:35.612429Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pzkshv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T06:58:22.354401Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T06:58:22.352715Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T06:58:22.352963Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T06:58:22.354635Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T06:58:22.353194Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T06:58:22.353422Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T06:58:22.798594Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:09:18.411 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:52 smithi193 bash[28559]: cluster 2023-11-17T07:07:51.625122+0000 mgr.smithi097.swvonl (mgr.14180) 850 : cluster [DBG] pgmap v625: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:54 smithi193 bash[28559]: cluster 2023-11-17T07:07:53.626273+0000 mgr.smithi097.swvonl (mgr.14180) 851 : cluster [DBG] pgmap v626: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:56 smithi193 bash[28559]: cluster 2023-11-17T07:07:55.627675+0000 mgr.smithi097.swvonl (mgr.14180) 852 : cluster [DBG] pgmap v627: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:56 smithi193 bash[28559]: audit 2023-11-17T07:07:55.681707+0000 mgr.smithi097.swvonl (mgr.14180) 853 : audit [DBG] from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:07:58 smithi193 bash[28559]: cluster 2023-11-17T07:07:57.628698+0000 mgr.smithi097.swvonl (mgr.14180) 854 : cluster [DBG] pgmap v628: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:00 smithi193 bash[28559]: cluster 2023-11-17T07:07:59.629761+0000 mgr.smithi097.swvonl (mgr.14180) 855 : cluster [DBG] pgmap v629: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:02 smithi193 bash[28559]: cluster 2023-11-17T07:08:01.630943+0000 mgr.smithi097.swvonl (mgr.14180) 856 : cluster [DBG] pgmap v630: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:04 smithi193 bash[28559]: cluster 2023-11-17T07:08:03.632048+0000 mgr.smithi097.swvonl (mgr.14180) 857 : cluster [DBG] pgmap v631: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:06 smithi193 bash[28559]: cluster 2023-11-17T07:08:05.633465+0000 mgr.smithi097.swvonl (mgr.14180) 858 : cluster [DBG] pgmap v632: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:08 smithi193 bash[28559]: cluster 2023-11-17T07:08:07.634452+0000 mgr.smithi097.swvonl (mgr.14180) 859 : cluster [DBG] pgmap v633: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:10 smithi193 bash[28559]: cluster 2023-11-17T07:08:09.635486+0000 mgr.smithi097.swvonl (mgr.14180) 860 : cluster [DBG] pgmap v634: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:12 smithi193 bash[28559]: cluster 2023-11-17T07:08:11.636720+0000 mgr.smithi097.swvonl (mgr.14180) 861 : cluster [DBG] pgmap v635: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:14 smithi193 bash[28559]: cluster 2023-11-17T07:08:13.637819+0000 mgr.smithi097.swvonl (mgr.14180) 862 : cluster [DBG] pgmap v636: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:16 smithi193 bash[28559]: cluster 2023-11-17T07:08:15.639194+0000 mgr.smithi097.swvonl (mgr.14180) 863 : cluster [DBG] pgmap v637: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:18 smithi193 bash[28559]: cluster 2023-11-17T07:08:17.640189+0000 mgr.smithi097.swvonl (mgr.14180) 864 : cluster [DBG] pgmap v638: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:20 smithi193 bash[28559]: cluster 2023-11-17T07:08:19.641269+0000 mgr.smithi097.swvonl (mgr.14180) 865 : cluster [DBG] pgmap v639: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:22 smithi193 bash[28559]: cluster 2023-11-17T07:08:21.642508+0000 mgr.smithi097.swvonl (mgr.14180) 866 : cluster [DBG] pgmap v640: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:24 smithi193 bash[28559]: cluster 2023-11-17T07:08:23.643539+0000 mgr.smithi097.swvonl (mgr.14180) 867 : cluster [DBG] pgmap v641: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:26 smithi193 bash[28559]: cluster 2023-11-17T07:08:25.644869+0000 mgr.smithi097.swvonl (mgr.14180) 868 : cluster [DBG] pgmap v642: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:28 smithi193 bash[28559]: cluster 2023-11-17T07:08:27.645887+0000 mgr.smithi097.swvonl (mgr.14180) 869 : cluster [DBG] pgmap v643: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:30 smithi193 bash[28559]: cluster 2023-11-17T07:08:29.647099+0000 mgr.smithi097.swvonl (mgr.14180) 870 : cluster [DBG] pgmap v644: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:32 smithi193 bash[28559]: cluster 2023-11-17T07:08:31.648366+0000 mgr.smithi097.swvonl (mgr.14180) 871 : cluster [DBG] pgmap v645: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:34 smithi193 bash[28559]: cluster 2023-11-17T07:08:33.649426+0000 mgr.smithi097.swvonl (mgr.14180) 872 : cluster [DBG] pgmap v646: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:35 smithi193 bash[28559]: audit 2023-11-17T07:08:35.619653+0000 mon.smithi097 (mon.0) 886 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:09:18.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:36 smithi193 bash[28559]: cluster 2023-11-17T07:08:35.650670+0000 mgr.smithi097.swvonl (mgr.14180) 873 : cluster [DBG] pgmap v647: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:38 smithi193 bash[28559]: cluster 2023-11-17T07:08:37.651661+0000 mgr.smithi097.swvonl (mgr.14180) 874 : cluster [DBG] pgmap v648: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:40 smithi193 bash[28559]: cluster 2023-11-17T07:08:39.652844+0000 mgr.smithi097.swvonl (mgr.14180) 875 : cluster [DBG] pgmap v649: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.460 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:41.385059+0000 mon.smithi097 (mon.0) 887 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: cluster 2023-11-17T07:08:41.653810+0000 mgr.smithi097.swvonl (mgr.14180) 876 : cluster [DBG] pgmap v650: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:41.854901+0000 mon.smithi097 (mon.0) 888 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:42.157786+0000 mon.smithi097 (mon.0) 889 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:42.160055+0000 mon.smithi097 (mon.0) 890 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:42.176721+0000 mon.smithi097 (mon.0) 891 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:42.184987+0000 mon.smithi097 (mon.0) 892 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:42 smithi193 bash[28559]: audit 2023-11-17T07:08:42.199685+0000 mon.smithi097 (mon.0) 893 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:43 smithi193 bash[28559]: cluster 2023-11-17T07:08:42.177437+0000 mgr.smithi097.swvonl (mgr.14180) 877 : cluster [DBG] pgmap v651: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:43 smithi193 bash[28559]: cephadm 2023-11-17T07:08:42.203891+0000 mgr.smithi097.swvonl (mgr.14180) 878 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.aehjpz on smithi193 2023-11-17T07:09:18.461 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:43 smithi193 bash[28559]: cluster 2023-11-17T07:08:42.407054+0000 mon.smithi097 (mon.0) 894 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:43 smithi193 bash[28559]: cluster 2023-11-17T07:08:42.407104+0000 mon.smithi097 (mon.0) 895 : cluster [INF] Cluster is now healthy 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:45 smithi193 bash[28559]: cluster 2023-11-17T07:08:44.178211+0000 mgr.smithi097.swvonl (mgr.14180) 879 : cluster [DBG] pgmap v652: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:46 smithi193 bash[28559]: audit 2023-11-17T07:08:45.959507+0000 mon.smithi097 (mon.0) 896 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.aehjpz"}]: dispatch 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: cephadm 2023-11-17T07:08:45.958305+0000 mgr.smithi097.swvonl (mgr.14180) 880 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.462 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:18.463 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:09:18.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: cephadm 2023-11-17T07:08:45.958919+0000 mgr.smithi097.swvonl (mgr.14180) 881 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: cephadm 2023-11-17T07:08:45.961140+0000 mgr.smithi097.swvonl (mgr.14180) 882 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: cephadm 2023-11-17T07:08:45.966023+0000 mgr.smithi097.swvonl (mgr.14180) 883 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vkvbmr on smithi097 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:47 smithi193 bash[28559]: cluster 2023-11-17T07:08:46.178830+0000 mgr.smithi097.swvonl (mgr.14180) 884 : cluster [DBG] pgmap v653: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:49 smithi193 bash[28559]: cluster 2023-11-17T07:08:48.179572+0000 mgr.smithi097.swvonl (mgr.14180) 885 : cluster [DBG] pgmap v654: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cephadm 2023-11-17T07:08:49.722563+0000 mgr.smithi097.swvonl (mgr.14180) 886 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:09:18.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cephadm 2023-11-17T07:08:49.723114+0000 mgr.smithi097.swvonl (mgr.14180) 887 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: audit 2023-11-17T07:08:49.723783+0000 mon.smithi097 (mon.0) 897 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vkvbmr"}]: dispatch 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cephadm 2023-11-17T07:08:49.725199+0000 mgr.smithi097.swvonl (mgr.14180) 888 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.515 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cephadm 2023-11-17T07:08:49.728293+0000 mgr.smithi097.swvonl (mgr.14180) 889 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:18.515 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cephadm 2023-11-17T07:08:49.731047+0000 mgr.smithi097.swvonl (mgr.14180) 890 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:18.515 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:50 smithi193 bash[28559]: cluster 2023-11-17T07:08:49.732487+0000 mgr.smithi097.swvonl (mgr.14180) 891 : cluster [DBG] pgmap v655: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2023-11-17T07:09:18.515 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:51 smithi193 bash[28559]: cluster 2023-11-17T07:08:50.419363+0000 mon.smithi097 (mon.0) 898 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:09:18.515 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:52 smithi193 bash[28559]: cluster 2023-11-17T07:08:51.733664+0000 mgr.smithi097.swvonl (mgr.14180) 892 : cluster [DBG] pgmap v656: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:54 smithi193 bash[28559]: cluster 2023-11-17T07:08:53.734816+0000 mgr.smithi097.swvonl (mgr.14180) 893 : cluster [DBG] pgmap v657: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:54 smithi193 bash[28559]: audit 2023-11-17T07:08:53.885438+0000 mon.smithi097 (mon.0) 899 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:56 smithi193 bash[28559]: cluster 2023-11-17T07:08:55.736168+0000 mgr.smithi097.swvonl (mgr.14180) 894 : cluster [DBG] pgmap v658: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:08:58 smithi193 bash[28559]: cluster 2023-11-17T07:08:57.737181+0000 mgr.smithi097.swvonl (mgr.14180) 895 : cluster [DBG] pgmap v659: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:00 smithi193 bash[28559]: cluster 2023-11-17T07:08:59.738465+0000 mgr.smithi097.swvonl (mgr.14180) 896 : cluster [DBG] pgmap v660: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 354 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:02 smithi193 bash[28559]: cluster 2023-11-17T07:09:01.739572+0000 mgr.smithi097.swvonl (mgr.14180) 897 : cluster [DBG] pgmap v661: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:04 smithi193 bash[28559]: cluster 2023-11-17T07:09:03.740649+0000 mgr.smithi097.swvonl (mgr.14180) 898 : cluster [DBG] pgmap v662: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:06 smithi193 bash[28559]: cluster 2023-11-17T07:09:05.741933+0000 mgr.smithi097.swvonl (mgr.14180) 899 : cluster [DBG] pgmap v663: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:08 smithi193 bash[28559]: cluster 2023-11-17T07:09:07.742901+0000 mgr.smithi097.swvonl (mgr.14180) 900 : cluster [DBG] pgmap v664: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:10 smithi193 bash[28559]: cluster 2023-11-17T07:09:09.744175+0000 mgr.smithi097.swvonl (mgr.14180) 901 : cluster [DBG] pgmap v665: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.516 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:12 smithi193 bash[28559]: cluster 2023-11-17T07:09:11.745314+0000 mgr.smithi097.swvonl (mgr.14180) 902 : cluster [DBG] pgmap v666: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.517 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:14 smithi193 bash[28559]: cluster 2023-11-17T07:09:13.746469+0000 mgr.smithi097.swvonl (mgr.14180) 903 : cluster [DBG] pgmap v667: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.517 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:16 smithi193 bash[28559]: cluster 2023-11-17T07:09:15.747909+0000 mgr.smithi097.swvonl (mgr.14180) 904 : cluster [DBG] pgmap v668: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:56 smithi097 bash[24958]: cluster 2023-11-17T07:07:55.627675+0000 mgr.smithi097.swvonl (mgr.14180) 852 : cluster [DBG] pgmap v627: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:56 smithi097 bash[24958]: audit 2023-11-17T07:07:55.681707+0000 mgr.smithi097.swvonl (mgr.14180) 853 : audit [DBG] from='client.14872 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:07:58 smithi097 bash[24958]: cluster 2023-11-17T07:07:57.628698+0000 mgr.smithi097.swvonl (mgr.14180) 854 : cluster [DBG] pgmap v628: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:00 smithi097 bash[24958]: cluster 2023-11-17T07:07:59.629761+0000 mgr.smithi097.swvonl (mgr.14180) 855 : cluster [DBG] pgmap v629: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:02 smithi097 bash[24958]: cluster 2023-11-17T07:08:01.630943+0000 mgr.smithi097.swvonl (mgr.14180) 856 : cluster [DBG] pgmap v630: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:04 smithi097 bash[24958]: cluster 2023-11-17T07:08:03.632048+0000 mgr.smithi097.swvonl (mgr.14180) 857 : cluster [DBG] pgmap v631: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:06 smithi097 bash[24958]: cluster 2023-11-17T07:08:05.633465+0000 mgr.smithi097.swvonl (mgr.14180) 858 : cluster [DBG] pgmap v632: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:08 smithi097 bash[24958]: cluster 2023-11-17T07:08:07.634452+0000 mgr.smithi097.swvonl (mgr.14180) 859 : cluster [DBG] pgmap v633: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:10 smithi097 bash[24958]: cluster 2023-11-17T07:08:09.635486+0000 mgr.smithi097.swvonl (mgr.14180) 860 : cluster [DBG] pgmap v634: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:12 smithi097 bash[24958]: cluster 2023-11-17T07:08:11.636720+0000 mgr.smithi097.swvonl (mgr.14180) 861 : cluster [DBG] pgmap v635: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:14 smithi097 bash[24958]: cluster 2023-11-17T07:08:13.637819+0000 mgr.smithi097.swvonl (mgr.14180) 862 : cluster [DBG] pgmap v636: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:16 smithi097 bash[24958]: cluster 2023-11-17T07:08:15.639194+0000 mgr.smithi097.swvonl (mgr.14180) 863 : cluster [DBG] pgmap v637: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:18 smithi097 bash[24958]: cluster 2023-11-17T07:08:17.640189+0000 mgr.smithi097.swvonl (mgr.14180) 864 : cluster [DBG] pgmap v638: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:20 smithi097 bash[24958]: cluster 2023-11-17T07:08:19.641269+0000 mgr.smithi097.swvonl (mgr.14180) 865 : cluster [DBG] pgmap v639: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:22 smithi097 bash[24958]: cluster 2023-11-17T07:08:21.642508+0000 mgr.smithi097.swvonl (mgr.14180) 866 : cluster [DBG] pgmap v640: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:24 smithi097 bash[24958]: cluster 2023-11-17T07:08:23.643539+0000 mgr.smithi097.swvonl (mgr.14180) 867 : cluster [DBG] pgmap v641: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:26 smithi097 bash[24958]: cluster 2023-11-17T07:08:25.644869+0000 mgr.smithi097.swvonl (mgr.14180) 868 : cluster [DBG] pgmap v642: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:28 smithi097 bash[24958]: cluster 2023-11-17T07:08:27.645887+0000 mgr.smithi097.swvonl (mgr.14180) 869 : cluster [DBG] pgmap v643: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:30 smithi097 bash[24958]: cluster 2023-11-17T07:08:29.647099+0000 mgr.smithi097.swvonl (mgr.14180) 870 : cluster [DBG] pgmap v644: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:32 smithi097 bash[24958]: cluster 2023-11-17T07:08:31.648366+0000 mgr.smithi097.swvonl (mgr.14180) 871 : cluster [DBG] pgmap v645: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:34 smithi097 bash[24958]: cluster 2023-11-17T07:08:33.649426+0000 mgr.smithi097.swvonl (mgr.14180) 872 : cluster [DBG] pgmap v646: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:35 smithi097 bash[24958]: audit 2023-11-17T07:08:35.619653+0000 mon.smithi097 (mon.0) 886 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:09:18.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:36 smithi097 bash[24958]: cluster 2023-11-17T07:08:35.650670+0000 mgr.smithi097.swvonl (mgr.14180) 873 : cluster [DBG] pgmap v647: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:38 smithi097 bash[24958]: cluster 2023-11-17T07:08:37.651661+0000 mgr.smithi097.swvonl (mgr.14180) 874 : cluster [DBG] pgmap v648: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:40 smithi097 bash[24958]: cluster 2023-11-17T07:08:39.652844+0000 mgr.smithi097.swvonl (mgr.14180) 875 : cluster [DBG] pgmap v649: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:41.385059+0000 mon.smithi097 (mon.0) 887 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: cluster 2023-11-17T07:08:41.653810+0000 mgr.smithi097.swvonl (mgr.14180) 876 : cluster [DBG] pgmap v650: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:41.854901+0000 mon.smithi097 (mon.0) 888 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:42.157786+0000 mon.smithi097 (mon.0) 889 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:42.160055+0000 mon.smithi097 (mon.0) 890 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:09:18.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:42.176721+0000 mon.smithi097 (mon.0) 891 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:42.184987+0000 mon.smithi097 (mon.0) 892 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:42 smithi097 bash[24958]: audit 2023-11-17T07:08:42.199685+0000 mon.smithi097 (mon.0) 893 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:43 smithi097 bash[24958]: cluster 2023-11-17T07:08:42.177437+0000 mgr.smithi097.swvonl (mgr.14180) 877 : cluster [DBG] pgmap v651: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:43 smithi097 bash[24958]: cephadm 2023-11-17T07:08:42.203891+0000 mgr.smithi097.swvonl (mgr.14180) 878 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.aehjpz on smithi193 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:43 smithi097 bash[24958]: cluster 2023-11-17T07:08:42.407054+0000 mon.smithi097 (mon.0) 894 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:43 smithi097 bash[24958]: cluster 2023-11-17T07:08:42.407104+0000 mon.smithi097 (mon.0) 895 : cluster [INF] Cluster is now healthy 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:45 smithi097 bash[24958]: cluster 2023-11-17T07:08:44.178211+0000 mgr.smithi097.swvonl (mgr.14180) 879 : cluster [DBG] pgmap v652: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:46 smithi097 bash[24958]: audit 2023-11-17T07:08:45.959507+0000 mon.smithi097 (mon.0) 896 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.aehjpz"}]: dispatch 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: cephadm 2023-11-17T07:08:45.958305+0000 mgr.smithi097.swvonl (mgr.14180) 880 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:09:18.522 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: cephadm 2023-11-17T07:08:45.958919+0000 mgr.smithi097.swvonl (mgr.14180) 881 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: cephadm 2023-11-17T07:08:45.961140+0000 mgr.smithi097.swvonl (mgr.14180) 882 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.aehjpz ... 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: cephadm 2023-11-17T07:08:45.966023+0000 mgr.smithi097.swvonl (mgr.14180) 883 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vkvbmr on smithi097 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:47 smithi097 bash[24958]: cluster 2023-11-17T07:08:46.178830+0000 mgr.smithi097.swvonl (mgr.14180) 884 : cluster [DBG] pgmap v653: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:49 smithi097 bash[24958]: cluster 2023-11-17T07:08:48.179572+0000 mgr.smithi097.swvonl (mgr.14180) 885 : cluster [DBG] pgmap v654: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cephadm 2023-11-17T07:08:49.722563+0000 mgr.smithi097.swvonl (mgr.14180) 886 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:09:18.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cephadm 2023-11-17T07:08:49.723114+0000 mgr.smithi097.swvonl (mgr.14180) 887 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: audit 2023-11-17T07:08:49.723783+0000 mon.smithi097 (mon.0) 897 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vkvbmr"}]: dispatch 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cephadm 2023-11-17T07:08:49.725199+0000 mgr.smithi097.swvonl (mgr.14180) 888 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vkvbmr ... 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cephadm 2023-11-17T07:08:49.728293+0000 mgr.smithi097.swvonl (mgr.14180) 889 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cephadm 2023-11-17T07:08:49.731047+0000 mgr.smithi097.swvonl (mgr.14180) 890 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:50 smithi097 bash[24958]: cluster 2023-11-17T07:08:49.732487+0000 mgr.smithi097.swvonl (mgr.14180) 891 : cluster [DBG] pgmap v655: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:51 smithi097 bash[24958]: cluster 2023-11-17T07:08:50.419363+0000 mon.smithi097 (mon.0) 898 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:52 smithi097 bash[24958]: cluster 2023-11-17T07:08:51.733664+0000 mgr.smithi097.swvonl (mgr.14180) 892 : cluster [DBG] pgmap v656: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:54 smithi097 bash[24958]: cluster 2023-11-17T07:08:53.734816+0000 mgr.smithi097.swvonl (mgr.14180) 893 : cluster [DBG] pgmap v657: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:54 smithi097 bash[24958]: audit 2023-11-17T07:08:53.885438+0000 mon.smithi097 (mon.0) 899 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:18.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:56 smithi097 bash[24958]: cluster 2023-11-17T07:08:55.736168+0000 mgr.smithi097.swvonl (mgr.14180) 894 : cluster [DBG] pgmap v658: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:08:58 smithi097 bash[24958]: cluster 2023-11-17T07:08:57.737181+0000 mgr.smithi097.swvonl (mgr.14180) 895 : cluster [DBG] pgmap v659: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:00 smithi097 bash[24958]: cluster 2023-11-17T07:08:59.738465+0000 mgr.smithi097.swvonl (mgr.14180) 896 : cluster [DBG] pgmap v660: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 354 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:02 smithi097 bash[24958]: cluster 2023-11-17T07:09:01.739572+0000 mgr.smithi097.swvonl (mgr.14180) 897 : cluster [DBG] pgmap v661: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:04 smithi097 bash[24958]: cluster 2023-11-17T07:09:03.740649+0000 mgr.smithi097.swvonl (mgr.14180) 898 : cluster [DBG] pgmap v662: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:06 smithi097 bash[24958]: cluster 2023-11-17T07:09:05.741933+0000 mgr.smithi097.swvonl (mgr.14180) 899 : cluster [DBG] pgmap v663: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:08 smithi097 bash[24958]: cluster 2023-11-17T07:09:07.742901+0000 mgr.smithi097.swvonl (mgr.14180) 900 : cluster [DBG] pgmap v664: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:10 smithi097 bash[24958]: cluster 2023-11-17T07:09:09.744175+0000 mgr.smithi097.swvonl (mgr.14180) 901 : cluster [DBG] pgmap v665: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:12 smithi097 bash[24958]: cluster 2023-11-17T07:09:11.745314+0000 mgr.smithi097.swvonl (mgr.14180) 902 : cluster [DBG] pgmap v666: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:14 smithi097 bash[24958]: cluster 2023-11-17T07:09:13.746469+0000 mgr.smithi097.swvonl (mgr.14180) 903 : cluster [DBG] pgmap v667: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:18.531 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:16 smithi097 bash[24958]: cluster 2023-11-17T07:09:15.747909+0000 mgr.smithi097.swvonl (mgr.14180) 904 : cluster [DBG] pgmap v668: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:19.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:18 smithi097 bash[24958]: cluster 2023-11-17T07:09:17.748874+0000 mgr.smithi097.swvonl (mgr.14180) 905 : cluster [DBG] pgmap v669: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:19.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:18 smithi193 bash[28559]: cluster 2023-11-17T07:09:17.748874+0000 mgr.smithi097.swvonl (mgr.14180) 905 : cluster [DBG] pgmap v669: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:19.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:09:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:20 smithi097 bash[24958]: cluster 2023-11-17T07:09:19.749805+0000 mgr.smithi097.swvonl (mgr.14180) 906 : cluster [DBG] pgmap v670: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:21.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:20 smithi193 bash[28559]: cluster 2023-11-17T07:09:19.749805+0000 mgr.smithi097.swvonl (mgr.14180) 906 : cluster [DBG] pgmap v670: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:22.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:22 smithi097 bash[24958]: cluster 2023-11-17T07:09:21.750571+0000 mgr.smithi097.swvonl (mgr.14180) 907 : cluster [DBG] pgmap v671: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:22.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:22 smithi193 bash[28559]: cluster 2023-11-17T07:09:21.750571+0000 mgr.smithi097.swvonl (mgr.14180) 907 : cluster [DBG] pgmap v671: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:23.240 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:09:23.240 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:45.960876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\nDeploy daemon haproxy.nfs.foo.smithi193.aehjpz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.725016Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\nDeploy daemon haproxy.nfs.foo.smithi097.vkvbmr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.728128Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:09:24.051 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:09:25.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:09:25.063 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:24 smithi097 bash[24958]: audit 2023-11-17T07:09:23.225273+0000 mgr.smithi097.swvonl (mgr.14180) 908 : audit [DBG] from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:25.063 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:24 smithi097 bash[24958]: cluster 2023-11-17T07:09:23.751557+0000 mgr.smithi097.swvonl (mgr.14180) 909 : cluster [DBG] pgmap v672: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:25.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:24 smithi193 bash[28559]: audit 2023-11-17T07:09:23.225273+0000 mgr.smithi097.swvonl (mgr.14180) 908 : audit [DBG] from='client.14876 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:25.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:24 smithi193 bash[28559]: cluster 2023-11-17T07:09:23.751557+0000 mgr.smithi097.swvonl (mgr.14180) 909 : cluster [DBG] pgmap v672: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:27.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:26 smithi097 bash[24958]: cluster 2023-11-17T07:09:25.753216+0000 mgr.smithi097.swvonl (mgr.14180) 910 : cluster [DBG] pgmap v673: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:27.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:26 smithi193 bash[28559]: cluster 2023-11-17T07:09:25.753216+0000 mgr.smithi097.swvonl (mgr.14180) 910 : cluster [DBG] pgmap v673: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:28.562 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:09:28.562 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:45.960876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\nDeploy daemon haproxy.nfs.foo.smithi193.aehjpz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.725016Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\nDeploy daemon haproxy.nfs.foo.smithi097.vkvbmr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.728128Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:09:29.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:28 smithi097 bash[24958]: cluster 2023-11-17T07:09:27.754304+0000 mgr.smithi097.swvonl (mgr.14180) 911 : cluster [DBG] pgmap v674: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:29.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:28 smithi193 bash[28559]: cluster 2023-11-17T07:09:27.754304+0000 mgr.smithi097.swvonl (mgr.14180) 911 : cluster [DBG] pgmap v674: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.742 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:29 smithi193 bash[28559]: audit 2023-11-17T07:09:28.549014+0000 mgr.smithi097.swvonl (mgr.14180) 912 : audit [DBG] from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:30 smithi193 bash[28559]: cluster 2023-11-17T07:09:29.755559+0000 mgr.smithi097.swvonl (mgr.14180) 913 : cluster [DBG] pgmap v675: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:32 smithi193 bash[28559]: cluster 2023-11-17T07:09:31.756693+0000 mgr.smithi097.swvonl (mgr.14180) 914 : cluster [DBG] pgmap v676: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:34 smithi193 bash[28559]: cluster 2023-11-17T07:09:33.757716+0000 mgr.smithi097.swvonl (mgr.14180) 915 : cluster [DBG] pgmap v677: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:36 smithi193 bash[28559]: cluster 2023-11-17T07:09:35.759098+0000 mgr.smithi097.swvonl (mgr.14180) 916 : cluster [DBG] pgmap v678: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:38 smithi193 bash[28559]: cluster 2023-11-17T07:09:37.760068+0000 mgr.smithi097.swvonl (mgr.14180) 917 : cluster [DBG] pgmap v679: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:40 smithi193 bash[28559]: cluster 2023-11-17T07:09:39.761336+0000 mgr.smithi097.swvonl (mgr.14180) 918 : cluster [DBG] pgmap v680: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:42 smithi193 bash[28559]: cluster 2023-11-17T07:09:41.762447+0000 mgr.smithi097.swvonl (mgr.14180) 919 : cluster [DBG] pgmap v681: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.743 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:44 smithi193 bash[28559]: cluster 2023-11-17T07:09:43.763476+0000 mgr.smithi097.swvonl (mgr.14180) 920 : cluster [DBG] pgmap v682: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:29 smithi097 bash[24958]: audit 2023-11-17T07:09:28.549014+0000 mgr.smithi097.swvonl (mgr.14180) 912 : audit [DBG] from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:30 smithi097 bash[24958]: cluster 2023-11-17T07:09:29.755559+0000 mgr.smithi097.swvonl (mgr.14180) 913 : cluster [DBG] pgmap v675: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:32 smithi097 bash[24958]: cluster 2023-11-17T07:09:31.756693+0000 mgr.smithi097.swvonl (mgr.14180) 914 : cluster [DBG] pgmap v676: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:34 smithi097 bash[24958]: cluster 2023-11-17T07:09:33.757716+0000 mgr.smithi097.swvonl (mgr.14180) 915 : cluster [DBG] pgmap v677: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:36 smithi097 bash[24958]: cluster 2023-11-17T07:09:35.759098+0000 mgr.smithi097.swvonl (mgr.14180) 916 : cluster [DBG] pgmap v678: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:38 smithi097 bash[24958]: cluster 2023-11-17T07:09:37.760068+0000 mgr.smithi097.swvonl (mgr.14180) 917 : cluster [DBG] pgmap v679: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:40 smithi097 bash[24958]: cluster 2023-11-17T07:09:39.761336+0000 mgr.smithi097.swvonl (mgr.14180) 918 : cluster [DBG] pgmap v680: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:42 smithi097 bash[24958]: cluster 2023-11-17T07:09:41.762447+0000 mgr.smithi097.swvonl (mgr.14180) 919 : cluster [DBG] pgmap v681: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:45.744 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:44 smithi097 bash[24958]: cluster 2023-11-17T07:09:43.763476+0000 mgr.smithi097.swvonl (mgr.14180) 920 : cluster [DBG] pgmap v682: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:46.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:09:47.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:46 smithi097 bash[24958]: cluster 2023-11-17T07:09:45.764712+0000 mgr.smithi097.swvonl (mgr.14180) 921 : cluster [DBG] pgmap v683: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:47.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:46 smithi193 bash[28559]: cluster 2023-11-17T07:09:45.764712+0000 mgr.smithi097.swvonl (mgr.14180) 921 : cluster [DBG] pgmap v683: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:49.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:48 smithi097 bash[24958]: cluster 2023-11-17T07:09:47.765743+0000 mgr.smithi097.swvonl (mgr.14180) 922 : cluster [DBG] pgmap v684: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:48 smithi193 bash[28559]: cluster 2023-11-17T07:09:47.765743+0000 mgr.smithi097.swvonl (mgr.14180) 922 : cluster [DBG] pgmap v684: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:09:50.260 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:09:50.261 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:07:35.615336Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hcrbzp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:45.960876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\nDeploy daemon haproxy.nfs.foo.smithi193.aehjpz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.725016Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\nDeploy daemon haproxy.nfs.foo.smithi097.vkvbmr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.728128Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:09:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:49 smithi097 bash[24958]: audit 2023-11-17T07:09:49.735233+0000 mon.smithi097 (mon.0) 900 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:09:50.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:49 smithi193 bash[28559]: audit 2023-11-17T07:09:49.735233+0000 mon.smithi097 (mon.0) 900 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:09:51.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: cluster 2023-11-17T07:09:49.766808+0000 mgr.smithi097.swvonl (mgr.14180) 923 : cluster [DBG] pgmap v685: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:51.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.010886+0000 mon.smithi097 (mon.0) 901 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:51.145 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.324269+0000 mon.smithi097 (mon.0) 902 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:09:51.145 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.325690+0000 mon.smithi097 (mon.0) 903 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:09:51.145 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.331629+0000 mon.smithi097 (mon.0) 904 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:51.145 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.339002+0000 mon.smithi097 (mon.0) 905 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:09:51.145 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:50 smithi097 bash[24958]: audit 2023-11-17T07:09:50.349989+0000 mon.smithi097 (mon.0) 906 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:51.146 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:09:51.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: cluster 2023-11-17T07:09:49.766808+0000 mgr.smithi097.swvonl (mgr.14180) 923 : cluster [DBG] pgmap v685: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.010886+0000 mon.smithi097 (mon.0) 901 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.324269+0000 mon.smithi097 (mon.0) 902 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.325690+0000 mon.smithi097 (mon.0) 903 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.331629+0000 mon.smithi097 (mon.0) 904 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.339002+0000 mon.smithi097 (mon.0) 905 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:09:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:50 smithi193 bash[28559]: audit 2023-11-17T07:09:50.349989+0000 mon.smithi097 (mon.0) 906 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:52.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:09:52.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:51 smithi097 bash[24958]: audit 2023-11-17T07:09:50.247794+0000 mgr.smithi097.swvonl (mgr.14180) 924 : audit [DBG] from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:52.161 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:51 smithi097 bash[24958]: cluster 2023-11-17T07:09:50.332525+0000 mgr.smithi097.swvonl (mgr.14180) 925 : cluster [DBG] pgmap v686: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:52.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:51 smithi097 bash[24958]: cephadm 2023-11-17T07:09:50.353421+0000 mgr.smithi097.swvonl (mgr.14180) 926 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.yrlrbk on smithi193 2023-11-17T07:09:52.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:51 smithi097 bash[24958]: cluster 2023-11-17T07:09:51.329721+0000 mon.smithi097 (mon.0) 907 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:09:52.162 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:51 smithi097 bash[24958]: cluster 2023-11-17T07:09:51.329783+0000 mon.smithi097 (mon.0) 908 : cluster [INF] Cluster is now healthy 2023-11-17T07:09:52.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:51 smithi193 bash[28559]: audit 2023-11-17T07:09:50.247794+0000 mgr.smithi097.swvonl (mgr.14180) 924 : audit [DBG] from='client.14884 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:51 smithi193 bash[28559]: cluster 2023-11-17T07:09:50.332525+0000 mgr.smithi097.swvonl (mgr.14180) 925 : cluster [DBG] pgmap v686: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:51 smithi193 bash[28559]: cephadm 2023-11-17T07:09:50.353421+0000 mgr.smithi097.swvonl (mgr.14180) 926 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.yrlrbk on smithi193 2023-11-17T07:09:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:51 smithi193 bash[28559]: cluster 2023-11-17T07:09:51.329721+0000 mon.smithi097 (mon.0) 907 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:09:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:51 smithi193 bash[28559]: cluster 2023-11-17T07:09:51.329783+0000 mon.smithi097 (mon.0) 908 : cluster [INF] Cluster is now healthy 2023-11-17T07:09:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:53 smithi097 bash[24958]: cluster 2023-11-17T07:09:52.332923+0000 mgr.smithi097.swvonl (mgr.14180) 927 : cluster [DBG] pgmap v687: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:54.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:53 smithi193 bash[28559]: cluster 2023-11-17T07:09:52.332923+0000 mgr.smithi097.swvonl (mgr.14180) 927 : cluster [DBG] pgmap v687: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:55.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:54 smithi193 bash[28559]: audit 2023-11-17T07:09:54.395811+0000 mon.smithi097 (mon.0) 909 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.yrlrbk"}]: dispatch 2023-11-17T07:09:55.294 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:54 smithi097 bash[24958]: audit 2023-11-17T07:09:54.395811+0000 mon.smithi097 (mon.0) 909 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.yrlrbk"}]: dispatch 2023-11-17T07:09:55.739 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:09:55.739 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:45.960876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.aehjpz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-aehjpz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.aehjpz\nDeploy daemon haproxy.nfs.foo.smithi193.aehjpz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.725016Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.vkvbmr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vkvbmr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vkvbmr\nDeploy daemon haproxy.nfs.foo.smithi097.vkvbmr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:08:49.728128Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.veanyh on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: cluster 2023-11-17T07:09:54.333778+0000 mgr.smithi097.swvonl (mgr.14180) 928 : cluster [DBG] pgmap v688: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: cephadm 2023-11-17T07:09:54.394486+0000 mgr.smithi097.swvonl (mgr.14180) 929 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.038 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:56.039 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.040 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: cephadm 2023-11-17T07:09:54.395140+0000 mgr.smithi097.swvonl (mgr.14180) 930 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: cephadm 2023-11-17T07:09:54.397321+0000 mgr.smithi097.swvonl (mgr.14180) 931 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.042 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:55 smithi097 bash[24958]: cephadm 2023-11-17T07:09:54.400671+0000 mgr.smithi097.swvonl (mgr.14180) 932 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aezrvt on smithi097 2023-11-17T07:09:56.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: cluster 2023-11-17T07:09:54.333778+0000 mgr.smithi097.swvonl (mgr.14180) 928 : cluster [DBG] pgmap v688: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:56.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: cephadm 2023-11-17T07:09:54.394486+0000 mgr.smithi097.swvonl (mgr.14180) 929 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: cephadm 2023-11-17T07:09:54.395140+0000 mgr.smithi097.swvonl (mgr.14180) 930 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: cephadm 2023-11-17T07:09:54.397321+0000 mgr.smithi097.swvonl (mgr.14180) 931 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yrlrbk ... 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:56.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:56.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:56.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:56.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:55 smithi193 bash[28559]: cephadm 2023-11-17T07:09:54.400671+0000 mgr.smithi097.swvonl (mgr.14180) 932 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aezrvt on smithi097 2023-11-17T07:09:56.681 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:09:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:56 smithi097 bash[24958]: audit 2023-11-17T07:09:55.725080+0000 mgr.smithi097.swvonl (mgr.14180) 933 : audit [DBG] from='client.14888 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:57.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:56 smithi097 bash[24958]: audit 2023-11-17T07:09:56.532114+0000 mon.smithi097 (mon.0) 910 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:56 smithi193 bash[28559]: audit 2023-11-17T07:09:55.725080+0000 mgr.smithi097.swvonl (mgr.14180) 933 : audit [DBG] from='client.14888 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:09:57.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:56 smithi193 bash[28559]: audit 2023-11-17T07:09:56.532114+0000 mon.smithi097 (mon.0) 910 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:09:57.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:09:58.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:57 smithi097 bash[24958]: cluster 2023-11-17T07:09:56.334932+0000 mgr.smithi097.swvonl (mgr.14180) 934 : cluster [DBG] pgmap v689: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:58.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:57 smithi193 bash[28559]: cluster 2023-11-17T07:09:56.334932+0000 mgr.smithi097.swvonl (mgr.14180) 934 : cluster [DBG] pgmap v689: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:59.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cluster 2023-11-17T07:09:58.335720+0000 mgr.smithi097.swvonl (mgr.14180) 935 : cluster [DBG] pgmap v690: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:59.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cephadm 2023-11-17T07:09:58.810784+0000 mgr.smithi097.swvonl (mgr.14180) 936 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:09:59.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cephadm 2023-11-17T07:09:58.811360+0000 mgr.smithi097.swvonl (mgr.14180) 937 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: audit 2023-11-17T07:09:58.811952+0000 mon.smithi097 (mon.0) 911 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aezrvt"}]: dispatch 2023-11-17T07:09:59.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cephadm 2023-11-17T07:09:58.813187+0000 mgr.smithi097.swvonl (mgr.14180) 938 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.055 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cephadm 2023-11-17T07:09:58.816267+0000 mgr.smithi097.swvonl (mgr.14180) 939 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:59.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cephadm 2023-11-17T07:09:58.819141+0000 mgr.smithi097.swvonl (mgr.14180) 940 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:59.056 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:58 smithi097 bash[24958]: cluster 2023-11-17T07:09:58.820178+0000 mgr.smithi097.swvonl (mgr.14180) 941 : cluster [DBG] pgmap v691: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cluster 2023-11-17T07:09:58.335720+0000 mgr.smithi097.swvonl (mgr.14180) 935 : cluster [DBG] pgmap v690: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cephadm 2023-11-17T07:09:58.810784+0000 mgr.smithi097.swvonl (mgr.14180) 936 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cephadm 2023-11-17T07:09:58.811360+0000 mgr.smithi097.swvonl (mgr.14180) 937 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: audit 2023-11-17T07:09:58.811952+0000 mon.smithi097 (mon.0) 911 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aezrvt"}]: dispatch 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cephadm 2023-11-17T07:09:58.813187+0000 mgr.smithi097.swvonl (mgr.14180) 938 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aezrvt ... 2023-11-17T07:09:59.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cephadm 2023-11-17T07:09:58.816267+0000 mgr.smithi097.swvonl (mgr.14180) 939 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cephadm 2023-11-17T07:09:58.819141+0000 mgr.smithi097.swvonl (mgr.14180) 940 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:09:59.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:58 smithi193 bash[28559]: cluster 2023-11-17T07:09:58.820178+0000 mgr.smithi097.swvonl (mgr.14180) 941 : cluster [DBG] pgmap v691: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:10:00.181 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:59 smithi097 bash[24958]: cluster 2023-11-17T07:09:58.880767+0000 mon.smithi097 (mon.0) 912 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:10:00.181 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:09:59 smithi097 bash[24958]: audit 2023-11-17T07:09:58.933579+0000 mon.smithi097 (mon.0) 913 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:10:00.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:59 smithi193 bash[28559]: cluster 2023-11-17T07:09:58.880767+0000 mon.smithi097 (mon.0) 912 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:10:00.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:09:59 smithi193 bash[28559]: audit 2023-11-17T07:09:58.933579+0000 mon.smithi097 (mon.0) 913 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:10:01.158 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:00 smithi097 bash[24958]: cluster 2023-11-17T07:10:00.000154+0000 mon.smithi097 (mon.0) 914 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:10:01.159 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:00 smithi097 bash[24958]: cluster 2023-11-17T07:10:00.821488+0000 mgr.smithi097.swvonl (mgr.14180) 942 : cluster [DBG] pgmap v692: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:01.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:00 smithi193 bash[28559]: cluster 2023-11-17T07:10:00.000154+0000 mon.smithi097 (mon.0) 914 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:10:01.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:00 smithi193 bash[28559]: cluster 2023-11-17T07:10:00.821488+0000 mgr.smithi097.swvonl (mgr.14180) 942 : cluster [DBG] pgmap v692: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:01.526 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:01.526 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:02.329 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:02.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:02 smithi097 bash[24958]: audit 2023-11-17T07:10:01.512812+0000 mgr.smithi097.swvonl (mgr.14180) 943 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:02.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:02 smithi193 bash[28559]: audit 2023-11-17T07:10:01.512812+0000 mgr.smithi097.swvonl (mgr.14180) 943 : audit [DBG] from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:03.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:03.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:03 smithi097 bash[24958]: cluster 2023-11-17T07:10:02.822478+0000 mgr.smithi097.swvonl (mgr.14180) 944 : cluster [DBG] pgmap v693: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:03.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:03 smithi193 bash[28559]: cluster 2023-11-17T07:10:02.822478+0000 mgr.smithi097.swvonl (mgr.14180) 944 : cluster [DBG] pgmap v693: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:06.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:05 smithi097 bash[24958]: cluster 2023-11-17T07:10:04.823256+0000 mgr.smithi097.swvonl (mgr.14180) 945 : cluster [DBG] pgmap v694: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:06.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:05 smithi193 bash[28559]: cluster 2023-11-17T07:10:04.823256+0000 mgr.smithi097.swvonl (mgr.14180) 945 : cluster [DBG] pgmap v694: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:07.027 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:07.027 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:07.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:07 smithi097 bash[24958]: cluster 2023-11-17T07:10:06.823779+0000 mgr.smithi097.swvonl (mgr.14180) 946 : cluster [DBG] pgmap v695: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:07 smithi193 bash[28559]: cluster 2023-11-17T07:10:06.823779+0000 mgr.smithi097.swvonl (mgr.14180) 946 : cluster [DBG] pgmap v695: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:07.902 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:08.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:08 smithi097 bash[24958]: audit 2023-11-17T07:10:07.012408+0000 mgr.smithi097.swvonl (mgr.14180) 947 : audit [DBG] from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:08.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:08 smithi193 bash[28559]: audit 2023-11-17T07:10:07.012408+0000 mgr.smithi097.swvonl (mgr.14180) 947 : audit [DBG] from='client.14896 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:08.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:09.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:09 smithi097 bash[24958]: cluster 2023-11-17T07:10:08.824897+0000 mgr.smithi097.swvonl (mgr.14180) 948 : cluster [DBG] pgmap v696: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:09.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:09 smithi193 bash[28559]: cluster 2023-11-17T07:10:08.824897+0000 mgr.smithi097.swvonl (mgr.14180) 948 : cluster [DBG] pgmap v696: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:10:12.147 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:11 smithi097 bash[24958]: cluster 2023-11-17T07:10:10.826231+0000 mgr.smithi097.swvonl (mgr.14180) 949 : cluster [DBG] pgmap v697: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:12.147 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:11 smithi097 bash[24958]: audit 2023-11-17T07:10:11.534164+0000 mon.smithi097 (mon.0) 915 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:10:12.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:11 smithi193 bash[28559]: cluster 2023-11-17T07:10:10.826231+0000 mgr.smithi097.swvonl (mgr.14180) 949 : cluster [DBG] pgmap v697: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:11 smithi193 bash[28559]: audit 2023-11-17T07:10:11.534164+0000 mon.smithi097 (mon.0) 915 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:10:12.522 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:12.523 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:13.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:12 smithi097 bash[24958]: audit 2023-11-17T07:10:12.513116+0000 mgr.smithi097.swvonl (mgr.14180) 950 : audit [DBG] from='client.14900 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:13.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:12 smithi097 bash[24958]: cluster 2023-11-17T07:10:12.826908+0000 mgr.smithi097.swvonl (mgr.14180) 951 : cluster [DBG] pgmap v698: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:12 smithi193 bash[28559]: audit 2023-11-17T07:10:12.513116+0000 mgr.smithi097.swvonl (mgr.14180) 950 : audit [DBG] from='client.14900 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:12 smithi193 bash[28559]: cluster 2023-11-17T07:10:12.826908+0000 mgr.smithi097.swvonl (mgr.14180) 951 : cluster [DBG] pgmap v698: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:13.418 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:14.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:16.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:15 smithi097 bash[24958]: cluster 2023-11-17T07:10:14.827745+0000 mgr.smithi097.swvonl (mgr.14180) 952 : cluster [DBG] pgmap v699: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:16.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:15 smithi193 bash[28559]: cluster 2023-11-17T07:10:14.827745+0000 mgr.smithi097.swvonl (mgr.14180) 952 : cluster [DBG] pgmap v699: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:17.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:17 smithi097 bash[24958]: cluster 2023-11-17T07:10:16.828577+0000 mgr.smithi097.swvonl (mgr.14180) 953 : cluster [DBG] pgmap v700: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:17.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:17 smithi193 bash[28559]: cluster 2023-11-17T07:10:16.828577+0000 mgr.smithi097.swvonl (mgr.14180) 953 : cluster [DBG] pgmap v700: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:18.065 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:18.065 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:18.854 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:19.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:18 smithi097 bash[24958]: audit 2023-11-17T07:10:18.057048+0000 mgr.smithi097.swvonl (mgr.14180) 954 : audit [DBG] from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:19.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:18 smithi097 bash[24958]: cluster 2023-11-17T07:10:18.829481+0000 mgr.smithi097.swvonl (mgr.14180) 955 : cluster [DBG] pgmap v701: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:19.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:18 smithi193 bash[28559]: audit 2023-11-17T07:10:18.057048+0000 mgr.smithi097.swvonl (mgr.14180) 954 : audit [DBG] from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:19.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:18 smithi193 bash[28559]: cluster 2023-11-17T07:10:18.829481+0000 mgr.smithi097.swvonl (mgr.14180) 955 : cluster [DBG] pgmap v701: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:19.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:22.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:21 smithi097 bash[24958]: cluster 2023-11-17T07:10:20.830231+0000 mgr.smithi097.swvonl (mgr.14180) 956 : cluster [DBG] pgmap v702: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:22.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:21 smithi193 bash[28559]: cluster 2023-11-17T07:10:20.830231+0000 mgr.smithi097.swvonl (mgr.14180) 956 : cluster [DBG] pgmap v702: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:23.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:23.280 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:23.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:22 smithi097 bash[24958]: cluster 2023-11-17T07:10:22.831025+0000 mgr.smithi097.swvonl (mgr.14180) 957 : cluster [DBG] pgmap v703: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:23.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:22 smithi193 bash[28559]: cluster 2023-11-17T07:10:22.831025+0000 mgr.smithi097.swvonl (mgr.14180) 957 : cluster [DBG] pgmap v703: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:24.152 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:23 smithi097 bash[24958]: audit 2023-11-17T07:10:23.271656+0000 mgr.smithi097.swvonl (mgr.14180) 958 : audit [DBG] from='client.14908 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:23 smithi193 bash[28559]: audit 2023-11-17T07:10:23.271656+0000 mgr.smithi097.swvonl (mgr.14180) 958 : audit [DBG] from='client.14908 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:25.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:25.165 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:24 smithi097 bash[24958]: cluster 2023-11-17T07:10:24.832480+0000 mgr.smithi097.swvonl (mgr.14180) 959 : cluster [DBG] pgmap v704: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:25.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:24 smithi193 bash[28559]: cluster 2023-11-17T07:10:24.832480+0000 mgr.smithi097.swvonl (mgr.14180) 959 : cluster [DBG] pgmap v704: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:27.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:27 smithi097 bash[24958]: cluster 2023-11-17T07:10:26.833961+0000 mgr.smithi097.swvonl (mgr.14180) 960 : cluster [DBG] pgmap v705: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:27.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:27 smithi193 bash[28559]: cluster 2023-11-17T07:10:26.833961+0000 mgr.smithi097.swvonl (mgr.14180) 960 : cluster [DBG] pgmap v705: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:28.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:28.435 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:29.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:28 smithi097 bash[24958]: audit 2023-11-17T07:10:28.422545+0000 mgr.smithi097.swvonl (mgr.14180) 961 : audit [DBG] from='client.14912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:29.205 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:28 smithi097 bash[24958]: cluster 2023-11-17T07:10:28.835145+0000 mgr.smithi097.swvonl (mgr.14180) 962 : cluster [DBG] pgmap v706: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:29.206 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:29.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:28 smithi193 bash[28559]: audit 2023-11-17T07:10:28.422545+0000 mgr.smithi097.swvonl (mgr.14180) 961 : audit [DBG] from='client.14912 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:29.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:28 smithi193 bash[28559]: cluster 2023-11-17T07:10:28.835145+0000 mgr.smithi097.swvonl (mgr.14180) 962 : cluster [DBG] pgmap v706: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:30.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:31.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:30 smithi097 bash[24958]: cluster 2023-11-17T07:10:30.836286+0000 mgr.smithi097.swvonl (mgr.14180) 963 : cluster [DBG] pgmap v707: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:31.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:30 smithi193 bash[28559]: cluster 2023-11-17T07:10:30.836286+0000 mgr.smithi097.swvonl (mgr.14180) 963 : cluster [DBG] pgmap v707: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:33.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:32 smithi097 bash[24958]: cluster 2023-11-17T07:10:32.837289+0000 mgr.smithi097.swvonl (mgr.14180) 964 : cluster [DBG] pgmap v708: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:33.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:32 smithi193 bash[28559]: cluster 2023-11-17T07:10:32.837289+0000 mgr.smithi097.swvonl (mgr.14180) 964 : cluster [DBG] pgmap v708: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:33.628 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:33.628 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:34.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:33 smithi097 bash[24958]: audit 2023-11-17T07:10:33.614498+0000 mgr.smithi097.swvonl (mgr.14180) 965 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:33 smithi193 bash[28559]: audit 2023-11-17T07:10:33.614498+0000 mgr.smithi097.swvonl (mgr.14180) 965 : audit [DBG] from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:34.467 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:34 smithi193 bash[28559]: cluster 2023-11-17T07:10:34.838115+0000 mgr.smithi097.swvonl (mgr.14180) 966 : cluster [DBG] pgmap v709: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:35.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:34 smithi097 bash[24958]: cluster 2023-11-17T07:10:34.838115+0000 mgr.smithi097.swvonl (mgr.14180) 966 : cluster [DBG] pgmap v709: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:35.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:37.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:37 smithi097 bash[24958]: cluster 2023-11-17T07:10:36.839031+0000 mgr.smithi097.swvonl (mgr.14180) 967 : cluster [DBG] pgmap v710: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:37.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:37 smithi193 bash[28559]: cluster 2023-11-17T07:10:36.839031+0000 mgr.smithi097.swvonl (mgr.14180) 967 : cluster [DBG] pgmap v710: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:39.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:38 smithi097 bash[24958]: cluster 2023-11-17T07:10:38.840002+0000 mgr.smithi097.swvonl (mgr.14180) 968 : cluster [DBG] pgmap v711: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:39.127 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:39.128 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:39.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:38 smithi193 bash[28559]: cluster 2023-11-17T07:10:38.840002+0000 mgr.smithi097.swvonl (mgr.14180) 968 : cluster [DBG] pgmap v711: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:39.876 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:40.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:39 smithi097 bash[24958]: audit 2023-11-17T07:10:39.113333+0000 mgr.smithi097.swvonl (mgr.14180) 969 : audit [DBG] from='client.14920 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:39 smithi193 bash[28559]: audit 2023-11-17T07:10:39.113333+0000 mgr.smithi097.swvonl (mgr.14180) 969 : audit [DBG] from='client.14920 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:40.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:41.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:40 smithi097 bash[24958]: cluster 2023-11-17T07:10:40.841319+0000 mgr.smithi097.swvonl (mgr.14180) 970 : cluster [DBG] pgmap v712: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:41.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:40 smithi193 bash[28559]: cluster 2023-11-17T07:10:40.841319+0000 mgr.smithi097.swvonl (mgr.14180) 970 : cluster [DBG] pgmap v712: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:43.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:42 smithi097 bash[24958]: cluster 2023-11-17T07:10:42.842216+0000 mgr.smithi097.swvonl (mgr.14180) 971 : cluster [DBG] pgmap v713: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:43.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:42 smithi193 bash[28559]: cluster 2023-11-17T07:10:42.842216+0000 mgr.smithi097.swvonl (mgr.14180) 971 : cluster [DBG] pgmap v713: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:44.481 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:44.481 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:45.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:44 smithi097 bash[24958]: audit 2023-11-17T07:10:44.468356+0000 mgr.smithi097.swvonl (mgr.14180) 972 : audit [DBG] from='client.14924 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:45.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:44 smithi097 bash[24958]: cluster 2023-11-17T07:10:44.843732+0000 mgr.smithi097.swvonl (mgr.14180) 973 : cluster [DBG] pgmap v714: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:45.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:44 smithi193 bash[28559]: audit 2023-11-17T07:10:44.468356+0000 mgr.smithi097.swvonl (mgr.14180) 972 : audit [DBG] from='client.14924 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:45.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:44 smithi193 bash[28559]: cluster 2023-11-17T07:10:44.843732+0000 mgr.smithi097.swvonl (mgr.14180) 973 : cluster [DBG] pgmap v714: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:45.298 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:46.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:47.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:47 smithi097 bash[24958]: cluster 2023-11-17T07:10:46.844696+0000 mgr.smithi097.swvonl (mgr.14180) 974 : cluster [DBG] pgmap v715: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:47.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:47 smithi193 bash[28559]: cluster 2023-11-17T07:10:46.844696+0000 mgr.smithi097.swvonl (mgr.14180) 974 : cluster [DBG] pgmap v715: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:49.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:48 smithi097 bash[24958]: cluster 2023-11-17T07:10:48.845803+0000 mgr.smithi097.swvonl (mgr.14180) 975 : cluster [DBG] pgmap v716: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:49.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:48 smithi193 bash[28559]: cluster 2023-11-17T07:10:48.845803+0000 mgr.smithi097.swvonl (mgr.14180) 975 : cluster [DBG] pgmap v716: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:49.762 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:49.763 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:50.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:49 smithi097 bash[24958]: audit 2023-11-17T07:10:49.748608+0000 mgr.smithi097.swvonl (mgr.14180) 976 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:50.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:49 smithi193 bash[28559]: audit 2023-11-17T07:10:49.748608+0000 mgr.smithi097.swvonl (mgr.14180) 976 : audit [DBG] from='client.14928 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:50.590 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:51.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:50 smithi097 bash[24958]: cluster 2023-11-17T07:10:50.847118+0000 mgr.smithi097.swvonl (mgr.14180) 977 : cluster [DBG] pgmap v717: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:51.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:50 smithi193 bash[28559]: cluster 2023-11-17T07:10:50.847118+0000 mgr.smithi097.swvonl (mgr.14180) 977 : cluster [DBG] pgmap v717: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:10:51.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:53.236 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:52 smithi097 bash[24958]: cluster 2023-11-17T07:10:52.848174+0000 mgr.smithi097.swvonl (mgr.14180) 978 : cluster [DBG] pgmap v718: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:53.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:52 smithi193 bash[28559]: cluster 2023-11-17T07:10:52.848174+0000 mgr.smithi097.swvonl (mgr.14180) 978 : cluster [DBG] pgmap v718: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:55.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:54 smithi097 bash[24958]: cluster 2023-11-17T07:10:54.849674+0000 mgr.smithi097.swvonl (mgr.14180) 979 : cluster [DBG] pgmap v719: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:10:55.221 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:10:55.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:54 smithi193 bash[28559]: cluster 2023-11-17T07:10:54.849674+0000 mgr.smithi097.swvonl (mgr.14180) 979 : cluster [DBG] pgmap v719: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:56.017 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:10:56.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:55 smithi097 bash[24958]: audit 2023-11-17T07:10:55.207974+0000 mgr.smithi097.swvonl (mgr.14180) 980 : audit [DBG] from='client.14932 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:56.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:55 smithi193 bash[28559]: audit 2023-11-17T07:10:55.207974+0000 mgr.smithi097.swvonl (mgr.14180) 980 : audit [DBG] from='client.14932 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:10:57.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:10:57.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:57 smithi097 bash[24958]: cluster 2023-11-17T07:10:56.850771+0000 mgr.smithi097.swvonl (mgr.14180) 981 : cluster [DBG] pgmap v720: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:57.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:57 smithi193 bash[28559]: cluster 2023-11-17T07:10:56.850771+0000 mgr.smithi097.swvonl (mgr.14180) 981 : cluster [DBG] pgmap v720: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:10:59.186 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:58 smithi097 bash[24958]: audit 2023-11-17T07:10:58.823537+0000 mon.smithi097 (mon.0) 916 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:10:59.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:58 smithi193 bash[28559]: audit 2023-11-17T07:10:58.823537+0000 mon.smithi097 (mon.0) 916 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: cluster 2023-11-17T07:10:58.851395+0000 mgr.smithi097.swvonl (mgr.14180) 982 : cluster [DBG] pgmap v721: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: audit 2023-11-17T07:10:59.126910+0000 mon.smithi097 (mon.0) 917 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: audit 2023-11-17T07:10:59.127823+0000 mon.smithi097 (mon.0) 918 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: audit 2023-11-17T07:10:59.133303+0000 mon.smithi097 (mon.0) 919 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: audit 2023-11-17T07:10:59.139438+0000 mon.smithi097 (mon.0) 920 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:11:00.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:10:59 smithi097 bash[24958]: audit 2023-11-17T07:10:59.152590+0000 mon.smithi097 (mon.0) 921 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:00.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: cluster 2023-11-17T07:10:58.851395+0000 mgr.smithi097.swvonl (mgr.14180) 982 : cluster [DBG] pgmap v721: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:00.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: audit 2023-11-17T07:10:59.126910+0000 mon.smithi097 (mon.0) 917 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:11:00.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: audit 2023-11-17T07:10:59.127823+0000 mon.smithi097 (mon.0) 918 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:11:00.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: audit 2023-11-17T07:10:59.133303+0000 mon.smithi097 (mon.0) 919 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:00.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: audit 2023-11-17T07:10:59.139438+0000 mon.smithi097 (mon.0) 920 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:11:00.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:10:59 smithi193 bash[28559]: audit 2023-11-17T07:10:59.152590+0000 mon.smithi097 (mon.0) 921 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:00.592 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:00.592 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:08:49.730899Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.infnir on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:01.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:00 smithi097 bash[24958]: cluster 2023-11-17T07:10:59.133843+0000 mgr.smithi097.swvonl (mgr.14180) 983 : cluster [DBG] pgmap v722: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:01.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:00 smithi097 bash[24958]: cephadm 2023-11-17T07:10:59.157748+0000 mgr.smithi097.swvonl (mgr.14180) 984 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kczkje on smithi193 2023-11-17T07:11:01.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:00 smithi097 bash[24958]: cluster 2023-11-17T07:11:00.131281+0000 mon.smithi097 (mon.0) 922 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:11:01.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:00 smithi097 bash[24958]: cluster 2023-11-17T07:11:00.131345+0000 mon.smithi097 (mon.0) 923 : cluster [INF] Cluster is now healthy 2023-11-17T07:11:01.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:00 smithi193 bash[28559]: cluster 2023-11-17T07:10:59.133843+0000 mgr.smithi097.swvonl (mgr.14180) 983 : cluster [DBG] pgmap v722: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:01.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:00 smithi193 bash[28559]: cephadm 2023-11-17T07:10:59.157748+0000 mgr.smithi097.swvonl (mgr.14180) 984 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kczkje on smithi193 2023-11-17T07:11:01.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:00 smithi193 bash[28559]: cluster 2023-11-17T07:11:00.131281+0000 mon.smithi097 (mon.0) 922 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:11:01.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:00 smithi193 bash[28559]: cluster 2023-11-17T07:11:00.131345+0000 mon.smithi097 (mon.0) 923 : cluster [INF] Cluster is now healthy 2023-11-17T07:11:01.507 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:02.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:01 smithi097 bash[24958]: audit 2023-11-17T07:11:00.580388+0000 mgr.smithi097.swvonl (mgr.14180) 985 : audit [DBG] from='client.14936 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:01 smithi193 bash[28559]: audit 2023-11-17T07:11:00.580388+0000 mgr.smithi097.swvonl (mgr.14180) 985 : audit [DBG] from='client.14936 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:02.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:02 smithi097 bash[24958]: cluster 2023-11-17T07:11:01.134840+0000 mgr.smithi097.swvonl (mgr.14180) 986 : cluster [DBG] pgmap v723: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: cephadm 2023-11-17T07:11:03.076474+0000 mgr.smithi097.swvonl (mgr.14180) 987 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.785 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.786 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: cephadm 2023-11-17T07:11:03.076973+0000 mgr.smithi097.swvonl (mgr.14180) 988 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: audit 2023-11-17T07:11:03.077685+0000 mon.smithi097 (mon.0) 924 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kczkje"}]: dispatch 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: cephadm 2023-11-17T07:11:03.079153+0000 mgr.smithi097.swvonl (mgr.14180) 989 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.787 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.788 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.832 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.832 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: cephadm 2023-11-17T07:11:03.082936+0000 mgr.smithi097.swvonl (mgr.14180) 990 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.bezpar on smithi097 2023-11-17T07:11:26.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:03 smithi097 bash[24958]: cluster 2023-11-17T07:11:03.135367+0000 mgr.smithi097.swvonl (mgr.14180) 991 : cluster [DBG] pgmap v724: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.833 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:26.834 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:54.397153Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yrlrbk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yrlrbk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yrlrbk\nDeploy daemon haproxy.nfs.foo.smithi193.yrlrbk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.812971Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aezrvt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aezrvt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aezrvt\nDeploy daemon haproxy.nfs.foo.smithi097.aezrvt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:09:58.816104Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zdrteq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:26.835 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:06 smithi097 bash[24958]: cluster 2023-11-17T07:11:05.136543+0000 mgr.smithi097.swvonl (mgr.14180) 992 : cluster [DBG] pgmap v725: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.837 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:26.838 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:02 smithi193 bash[28559]: cluster 2023-11-17T07:11:01.134840+0000 mgr.smithi097.swvonl (mgr.14180) 986 : cluster [DBG] pgmap v723: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: cephadm 2023-11-17T07:11:03.076474+0000 mgr.smithi097.swvonl (mgr.14180) 987 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.839 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:11:26.841 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:11:26.842 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:11:26.842 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.842 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.842 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: cephadm 2023-11-17T07:11:03.076973+0000 mgr.smithi097.swvonl (mgr.14180) 988 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: audit 2023-11-17T07:11:03.077685+0000 mon.smithi097 (mon.0) 924 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kczkje"}]: dispatch 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: cephadm 2023-11-17T07:11:03.079153+0000 mgr.smithi097.swvonl (mgr.14180) 989 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje 2023-11-17T07:11:26.844 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.845 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.845 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje 2023-11-17T07:11:26.845 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kczkje ... 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.846 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: cephadm 2023-11-17T07:11:03.082936+0000 mgr.smithi097.swvonl (mgr.14180) 990 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.bezpar on smithi097 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:03 smithi193 bash[28559]: cluster 2023-11-17T07:11:03.135367+0000 mgr.smithi097.swvonl (mgr.14180) 991 : cluster [DBG] pgmap v724: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:06 smithi193 bash[28559]: cluster 2023-11-17T07:11:05.136543+0000 mgr.smithi097.swvonl (mgr.14180) 992 : cluster [DBG] pgmap v725: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:07 smithi193 bash[28559]: audit 2023-11-17T07:11:06.074778+0000 mgr.smithi097.swvonl (mgr.14180) 993 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:07 smithi193 bash[28559]: audit 2023-11-17T07:11:07.056303+0000 mon.smithi097 (mon.0) 925 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.bezpar"}]: dispatch 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cephadm 2023-11-17T07:11:07.055575+0000 mgr.smithi097.swvonl (mgr.14180) 994 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.847 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:11:26.848 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:11:26.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:11:26.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:11:26.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.850 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cephadm 2023-11-17T07:11:07.055850+0000 mgr.smithi097.swvonl (mgr.14180) 995 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cephadm 2023-11-17T07:11:07.057410+0000 mgr.smithi097.swvonl (mgr.14180) 996 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.851 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cephadm 2023-11-17T07:11:07.059407+0000 mgr.smithi097.swvonl (mgr.14180) 997 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cephadm 2023-11-17T07:11:07.060985+0000 mgr.smithi097.swvonl (mgr.14180) 998 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:11:26.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cluster 2023-11-17T07:11:07.062020+0000 mgr.smithi097.swvonl (mgr.14180) 999 : cluster [DBG] pgmap v726: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:08 smithi193 bash[28559]: cluster 2023-11-17T07:11:07.452975+0000 mon.smithi097 (mon.0) 926 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:09 smithi193 bash[28559]: audit 2023-11-17T07:11:08.978767+0000 mon.smithi097 (mon.0) 927 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:09 smithi193 bash[28559]: cluster 2023-11-17T07:11:09.063119+0000 mgr.smithi097.swvonl (mgr.14180) 1000 : cluster [DBG] pgmap v727: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:12 smithi193 bash[28559]: cluster 2023-11-17T07:11:11.064497+0000 mgr.smithi097.swvonl (mgr.14180) 1001 : cluster [DBG] pgmap v728: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:14 smithi193 bash[28559]: cluster 2023-11-17T07:11:13.065463+0000 mgr.smithi097.swvonl (mgr.14180) 1002 : cluster [DBG] pgmap v729: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:16 smithi193 bash[28559]: cluster 2023-11-17T07:11:15.066813+0000 mgr.smithi097.swvonl (mgr.14180) 1003 : cluster [DBG] pgmap v730: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:18 smithi193 bash[28559]: cluster 2023-11-17T07:11:17.067860+0000 mgr.smithi097.swvonl (mgr.14180) 1004 : cluster [DBG] pgmap v731: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:20 smithi193 bash[28559]: cluster 2023-11-17T07:11:19.068949+0000 mgr.smithi097.swvonl (mgr.14180) 1005 : cluster [DBG] pgmap v732: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:26.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:22 smithi193 bash[28559]: cluster 2023-11-17T07:11:21.070341+0000 mgr.smithi097.swvonl (mgr.14180) 1006 : cluster [DBG] pgmap v733: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:26.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:24 smithi193 bash[28559]: cluster 2023-11-17T07:11:23.071292+0000 mgr.smithi097.swvonl (mgr.14180) 1007 : cluster [DBG] pgmap v734: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:26.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:26 smithi193 bash[28559]: cluster 2023-11-17T07:11:25.072554+0000 mgr.smithi097.swvonl (mgr.14180) 1008 : cluster [DBG] pgmap v735: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:26.857 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:07 smithi097 bash[24958]: audit 2023-11-17T07:11:06.074778+0000 mgr.smithi097.swvonl (mgr.14180) 993 : audit [DBG] from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:26.857 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:07 smithi097 bash[24958]: audit 2023-11-17T07:11:07.056303+0000 mon.smithi097 (mon.0) 925 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.bezpar"}]: dispatch 2023-11-17T07:11:26.857 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cephadm 2023-11-17T07:11:07.055575+0000 mgr.smithi097.swvonl (mgr.14180) 994 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.857 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.857 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.858 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:11:26.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cephadm 2023-11-17T07:11:07.055850+0000 mgr.smithi097.swvonl (mgr.14180) 995 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.861 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cephadm 2023-11-17T07:11:07.057410+0000 mgr.smithi097.swvonl (mgr.14180) 996 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.862 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.862 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar 2023-11-17T07:11:26.862 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.862 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:11:26.862 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bezpar ... 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:11:26.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cephadm 2023-11-17T07:11:07.059407+0000 mgr.smithi097.swvonl (mgr.14180) 997 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cephadm 2023-11-17T07:11:07.060985+0000 mgr.smithi097.swvonl (mgr.14180) 998 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cluster 2023-11-17T07:11:07.062020+0000 mgr.smithi097.swvonl (mgr.14180) 999 : cluster [DBG] pgmap v726: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:08 smithi097 bash[24958]: cluster 2023-11-17T07:11:07.452975+0000 mon.smithi097 (mon.0) 926 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:09 smithi097 bash[24958]: audit 2023-11-17T07:11:08.978767+0000 mon.smithi097 (mon.0) 927 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:11:26.864 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:09 smithi097 bash[24958]: cluster 2023-11-17T07:11:09.063119+0000 mgr.smithi097.swvonl (mgr.14180) 1000 : cluster [DBG] pgmap v727: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:11:26.865 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:12 smithi097 bash[24958]: cluster 2023-11-17T07:11:11.064497+0000 mgr.smithi097.swvonl (mgr.14180) 1001 : cluster [DBG] pgmap v728: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:11:26.865 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:14 smithi097 bash[24958]: cluster 2023-11-17T07:11:13.065463+0000 mgr.smithi097.swvonl (mgr.14180) 1002 : cluster [DBG] pgmap v729: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.865 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:16 smithi097 bash[24958]: cluster 2023-11-17T07:11:15.066813+0000 mgr.smithi097.swvonl (mgr.14180) 1003 : cluster [DBG] pgmap v730: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.865 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:18 smithi097 bash[24958]: cluster 2023-11-17T07:11:17.067860+0000 mgr.smithi097.swvonl (mgr.14180) 1004 : cluster [DBG] pgmap v731: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:11:26.866 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:20 smithi097 bash[24958]: cluster 2023-11-17T07:11:19.068949+0000 mgr.smithi097.swvonl (mgr.14180) 1005 : cluster [DBG] pgmap v732: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:26.866 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:22 smithi097 bash[24958]: cluster 2023-11-17T07:11:21.070341+0000 mgr.smithi097.swvonl (mgr.14180) 1006 : cluster [DBG] pgmap v733: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:26.866 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:24 smithi097 bash[24958]: cluster 2023-11-17T07:11:23.071292+0000 mgr.smithi097.swvonl (mgr.14180) 1007 : cluster [DBG] pgmap v734: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:26.866 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:26 smithi097 bash[24958]: cluster 2023-11-17T07:11:25.072554+0000 mgr.smithi097.swvonl (mgr.14180) 1008 : cluster [DBG] pgmap v735: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:27.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:28.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:28 smithi097 bash[24958]: cluster 2023-11-17T07:11:27.073627+0000 mgr.smithi097.swvonl (mgr.14180) 1009 : cluster [DBG] pgmap v736: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:28.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:28 smithi193 bash[28559]: cluster 2023-11-17T07:11:27.073627+0000 mgr.smithi097.swvonl (mgr.14180) 1009 : cluster [DBG] pgmap v736: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:30.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:30 smithi097 bash[24958]: cluster 2023-11-17T07:11:29.074697+0000 mgr.smithi097.swvonl (mgr.14180) 1010 : cluster [DBG] pgmap v737: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:30.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:30 smithi193 bash[28559]: cluster 2023-11-17T07:11:29.074697+0000 mgr.smithi097.swvonl (mgr.14180) 1010 : cluster [DBG] pgmap v737: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:31.762 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:31.762 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.057273Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\nDeploy daemon haproxy.nfs.foo.smithi097.bezpar ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.059308Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:07.060896Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:32.625 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:32 smithi097 bash[24958]: cluster 2023-11-17T07:11:31.076083+0000 mgr.smithi097.swvonl (mgr.14180) 1011 : cluster [DBG] pgmap v738: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:32 smithi097 bash[24958]: audit 2023-11-17T07:11:31.749777+0000 mgr.smithi097.swvonl (mgr.14180) 1012 : audit [DBG] from='client.14944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:32.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:32 smithi193 bash[28559]: cluster 2023-11-17T07:11:31.076083+0000 mgr.smithi097.swvonl (mgr.14180) 1011 : cluster [DBG] pgmap v738: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:32 smithi193 bash[28559]: audit 2023-11-17T07:11:31.749777+0000 mgr.smithi097.swvonl (mgr.14180) 1012 : audit [DBG] from='client.14944 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:33.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:34.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:34 smithi097 bash[24958]: cluster 2023-11-17T07:11:33.077087+0000 mgr.smithi097.swvonl (mgr.14180) 1013 : cluster [DBG] pgmap v739: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:34.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:34 smithi193 bash[28559]: cluster 2023-11-17T07:11:33.077087+0000 mgr.smithi097.swvonl (mgr.14180) 1013 : cluster [DBG] pgmap v739: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:36.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:36 smithi097 bash[24958]: cluster 2023-11-17T07:11:35.078365+0000 mgr.smithi097.swvonl (mgr.14180) 1014 : cluster [DBG] pgmap v740: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:36.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:36 smithi193 bash[28559]: cluster 2023-11-17T07:11:35.078365+0000 mgr.smithi097.swvonl (mgr.14180) 1014 : cluster [DBG] pgmap v740: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:37.275 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:37.276 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.057273Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\nDeploy daemon haproxy.nfs.foo.smithi097.bezpar ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.059308Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:07.060896Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:38.120 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:38.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:38 smithi097 bash[24958]: cluster 2023-11-17T07:11:37.079322+0000 mgr.smithi097.swvonl (mgr.14180) 1015 : cluster [DBG] pgmap v741: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:38.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:38 smithi097 bash[24958]: audit 2023-11-17T07:11:37.263212+0000 mgr.smithi097.swvonl (mgr.14180) 1016 : audit [DBG] from='client.14948 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:38.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:38 smithi193 bash[28559]: cluster 2023-11-17T07:11:37.079322+0000 mgr.smithi097.swvonl (mgr.14180) 1015 : cluster [DBG] pgmap v741: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:38.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:38 smithi193 bash[28559]: audit 2023-11-17T07:11:37.263212+0000 mgr.smithi097.swvonl (mgr.14180) 1016 : audit [DBG] from='client.14948 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:39.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:40.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:40 smithi097 bash[24958]: cluster 2023-11-17T07:11:39.080374+0000 mgr.smithi097.swvonl (mgr.14180) 1017 : cluster [DBG] pgmap v742: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:40.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:40 smithi193 bash[28559]: cluster 2023-11-17T07:11:39.080374+0000 mgr.smithi097.swvonl (mgr.14180) 1017 : cluster [DBG] pgmap v742: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:42.614 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:42.614 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.057273Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\nDeploy daemon haproxy.nfs.foo.smithi097.bezpar ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.059308Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:07.060896Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:42.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:42 smithi097 bash[24958]: cluster 2023-11-17T07:11:41.081637+0000 mgr.smithi097.swvonl (mgr.14180) 1018 : cluster [DBG] pgmap v743: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:42.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:42 smithi193 bash[28559]: cluster 2023-11-17T07:11:41.081637+0000 mgr.smithi097.swvonl (mgr.14180) 1018 : cluster [DBG] pgmap v743: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:43.530 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:43.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:43 smithi097 bash[24958]: audit 2023-11-17T07:11:42.600369+0000 mgr.smithi097.swvonl (mgr.14180) 1019 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:43.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:43 smithi193 bash[28559]: audit 2023-11-17T07:11:42.600369+0000 mgr.smithi097.swvonl (mgr.14180) 1019 : audit [DBG] from='client.14952 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:44.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:44.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:44 smithi097 bash[24958]: cluster 2023-11-17T07:11:43.082684+0000 mgr.smithi097.swvonl (mgr.14180) 1020 : cluster [DBG] pgmap v744: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:45.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:44 smithi193 bash[28559]: cluster 2023-11-17T07:11:43.082684+0000 mgr.smithi097.swvonl (mgr.14180) 1020 : cluster [DBG] pgmap v744: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:46.863 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:46 smithi097 bash[24958]: cluster 2023-11-17T07:11:45.083878+0000 mgr.smithi097.swvonl (mgr.14180) 1021 : cluster [DBG] pgmap v745: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.346 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:11:55.346 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.057273Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\nDeploy daemon haproxy.nfs.foo.smithi097.bezpar ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.059308Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:07.060896Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:11:55.346 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:48 smithi097 bash[24958]: cluster 2023-11-17T07:11:47.085074+0000 mgr.smithi097.swvonl (mgr.14180) 1022 : cluster [DBG] pgmap v746: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:46 smithi193 bash[28559]: cluster 2023-11-17T07:11:45.083878+0000 mgr.smithi097.swvonl (mgr.14180) 1021 : cluster [DBG] pgmap v745: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:48 smithi193 bash[28559]: cluster 2023-11-17T07:11:47.085074+0000 mgr.smithi097.swvonl (mgr.14180) 1022 : cluster [DBG] pgmap v746: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:49 smithi193 bash[28559]: audit 2023-11-17T07:11:48.322727+0000 mgr.smithi097.swvonl (mgr.14180) 1023 : audit [DBG] from='client.14956 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:55.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:50 smithi193 bash[28559]: cluster 2023-11-17T07:11:49.086273+0000 mgr.smithi097.swvonl (mgr.14180) 1024 : cluster [DBG] pgmap v747: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.347 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:52 smithi193 bash[28559]: cluster 2023-11-17T07:11:51.087574+0000 mgr.smithi097.swvonl (mgr.14180) 1025 : cluster [DBG] pgmap v748: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:55.348 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:54 smithi193 bash[28559]: cluster 2023-11-17T07:11:53.088649+0000 mgr.smithi097.swvonl (mgr.14180) 1026 : cluster [DBG] pgmap v749: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.348 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:11:55.352 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:49 smithi097 bash[24958]: audit 2023-11-17T07:11:48.322727+0000 mgr.smithi097.swvonl (mgr.14180) 1023 : audit [DBG] from='client.14956 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:11:55.352 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:50 smithi097 bash[24958]: cluster 2023-11-17T07:11:49.086273+0000 mgr.smithi097.swvonl (mgr.14180) 1024 : cluster [DBG] pgmap v747: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:55.352 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:52 smithi097 bash[24958]: cluster 2023-11-17T07:11:51.087574+0000 mgr.smithi097.swvonl (mgr.14180) 1025 : cluster [DBG] pgmap v748: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:11:55.352 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:54 smithi097 bash[24958]: cluster 2023-11-17T07:11:53.088649+0000 mgr.smithi097.swvonl (mgr.14180) 1026 : cluster [DBG] pgmap v749: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:56.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:11:56.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:56 smithi097 bash[24958]: cluster 2023-11-17T07:11:55.089988+0000 mgr.smithi097.swvonl (mgr.14180) 1027 : cluster [DBG] pgmap v750: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:56 smithi193 bash[28559]: cluster 2023-11-17T07:11:55.089988+0000 mgr.smithi097.swvonl (mgr.14180) 1027 : cluster [DBG] pgmap v750: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:59.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:11:58 smithi097 bash[24958]: cluster 2023-11-17T07:11:57.090871+0000 mgr.smithi097.swvonl (mgr.14180) 1028 : cluster [DBG] pgmap v751: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:11:59.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:11:58 smithi193 bash[28559]: cluster 2023-11-17T07:11:57.090871+0000 mgr.smithi097.swvonl (mgr.14180) 1028 : cluster [DBG] pgmap v751: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:12:00.114 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:08.016 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:09:58.818976Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbssns on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:03.078968Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kczkje on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kczkje\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kczkje\nDeploy daemon haproxy.nfs.foo.smithi193.kczkje ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.057273Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bezpar on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bezpar\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bezpar\nDeploy daemon haproxy.nfs.foo.smithi097.bezpar ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:11:07.059308Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.atiipu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:11:07.060896Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.bsbmfz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:00 smithi097 bash[24958]: cluster 2023-11-17T07:11:59.091922+0000 mgr.smithi097.swvonl (mgr.14180) 1029 : cluster [DBG] pgmap v752: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.053 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:08.094 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:00 smithi193 bash[28559]: cluster 2023-11-17T07:11:59.091922+0000 mgr.smithi097.swvonl (mgr.14180) 1029 : cluster [DBG] pgmap v752: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.094 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:01 smithi193 bash[28559]: audit 2023-11-17T07:12:00.102828+0000 mgr.smithi097.swvonl (mgr.14180) 1030 : audit [DBG] from='client.14960 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:08.094 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:02 smithi193 bash[28559]: cluster 2023-11-17T07:12:01.093226+0000 mgr.smithi097.swvonl (mgr.14180) 1031 : cluster [DBG] pgmap v753: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:04 smithi193 bash[28559]: cluster 2023-11-17T07:12:03.094201+0000 mgr.smithi097.swvonl (mgr.14180) 1032 : cluster [DBG] pgmap v754: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:06 smithi193 bash[28559]: cluster 2023-11-17T07:12:05.095649+0000 mgr.smithi097.swvonl (mgr.14180) 1033 : cluster [DBG] pgmap v755: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.064794+0000 mon.smithi097 (mon.0) 928 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.363558+0000 mon.smithi097 (mon.0) 929 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.364882+0000 mon.smithi097 (mon.0) 930 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.372787+0000 mon.smithi097 (mon.0) 931 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.380377+0000 mon.smithi097 (mon.0) 932 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:07 smithi193 bash[28559]: audit 2023-11-17T07:12:07.392122+0000 mon.smithi097 (mon.0) 933 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:08 smithi193 bash[28559]: cluster 2023-11-17T07:12:07.096493+0000 mgr.smithi097.swvonl (mgr.14180) 1034 : cluster [DBG] pgmap v756: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:08 smithi193 bash[28559]: cluster 2023-11-17T07:12:07.373581+0000 mgr.smithi097.swvonl (mgr.14180) 1035 : cluster [DBG] pgmap v757: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:08 smithi193 bash[28559]: cephadm 2023-11-17T07:12:07.396243+0000 mgr.smithi097.swvonl (mgr.14180) 1036 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.bbwtlw on smithi193 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:08 smithi193 bash[28559]: cluster 2023-11-17T07:12:07.631336+0000 mon.smithi097 (mon.0) 934 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:08.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:08 smithi193 bash[28559]: cluster 2023-11-17T07:12:07.631405+0000 mon.smithi097 (mon.0) 935 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:10 smithi193 bash[28559]: cluster 2023-11-17T07:12:09.374428+0000 mgr.smithi097.swvonl (mgr.14180) 1037 : cluster [DBG] pgmap v758: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:11 smithi193 bash[28559]: audit 2023-11-17T07:12:11.357754+0000 mon.smithi097 (mon.0) 936 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.bbwtlw"}]: dispatch 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:11 smithi193 bash[28559]: audit 2023-11-17T07:12:11.537495+0000 mon.smithi097 (mon.0) 937 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: cephadm 2023-11-17T07:12:11.356379+0000 mgr.smithi097.swvonl (mgr.14180) 1038 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: cephadm 2023-11-17T07:12:11.356927+0000 mgr.smithi097.swvonl (mgr.14180) 1039 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: cephadm 2023-11-17T07:12:11.359673+0000 mgr.smithi097.swvonl (mgr.14180) 1040 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.bbwtlw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: cephadm 2023-11-17T07:12:11.365150+0000 mgr.smithi097.swvonl (mgr.14180) 1041 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vvyuqu on smithi097 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:12 smithi193 bash[28559]: cluster 2023-11-17T07:12:11.375470+0000 mgr.smithi097.swvonl (mgr.14180) 1042 : cluster [DBG] pgmap v759: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:14 smithi193 bash[28559]: cluster 2023-11-17T07:12:13.376273+0000 mgr.smithi097.swvonl (mgr.14180) 1043 : cluster [DBG] pgmap v760: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:15 smithi193 bash[28559]: audit 2023-11-17T07:12:15.119366+0000 mon.smithi097 (mon.0) 938 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vvyuqu"}]: dispatch 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cephadm 2023-11-17T07:12:15.118269+0000 mgr.smithi097.swvonl (mgr.14180) 1044 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.119 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:08.120 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.121 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cephadm 2023-11-17T07:12:15.118808+0000 mgr.smithi097.swvonl (mgr.14180) 1045 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cephadm 2023-11-17T07:12:15.120694+0000 mgr.smithi097.swvonl (mgr.14180) 1046 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vvyuqu on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.122 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cephadm 2023-11-17T07:12:15.123749+0000 mgr.smithi097.swvonl (mgr.14180) 1047 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.qqpouv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cephadm 2023-11-17T07:12:15.126683+0000 mgr.smithi097.swvonl (mgr.14180) 1048 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.tqczco on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cluster 2023-11-17T07:12:15.128106+0000 mgr.smithi097.swvonl (mgr.14180) 1049 : cluster [DBG] pgmap v761: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:16 smithi193 bash[28559]: cluster 2023-11-17T07:12:15.728482+0000 mon.smithi097 (mon.0) 939 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:18 smithi193 bash[28559]: cluster 2023-11-17T07:12:17.129188+0000 mgr.smithi097.swvonl (mgr.14180) 1050 : cluster [DBG] pgmap v762: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:20 smithi193 bash[28559]: audit 2023-11-17T07:12:19.028544+0000 mon.smithi097 (mon.0) 940 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:20 smithi193 bash[28559]: cluster 2023-11-17T07:12:19.129855+0000 mgr.smithi097.swvonl (mgr.14180) 1051 : cluster [DBG] pgmap v763: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:22 smithi193 bash[28559]: cluster 2023-11-17T07:12:21.131152+0000 mgr.smithi097.swvonl (mgr.14180) 1052 : cluster [DBG] pgmap v764: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:24 smithi193 bash[28559]: cluster 2023-11-17T07:12:23.132049+0000 mgr.smithi097.swvonl (mgr.14180) 1053 : cluster [DBG] pgmap v765: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:26 smithi193 bash[28559]: cluster 2023-11-17T07:12:25.133437+0000 mgr.smithi097.swvonl (mgr.14180) 1054 : cluster [DBG] pgmap v766: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:27 smithi193 bash[28559]: audit 2023-11-17T07:12:26.535949+0000 mon.smithi097 (mon.0) 941 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:28 smithi193 bash[28559]: cluster 2023-11-17T07:12:27.134130+0000 mgr.smithi097.swvonl (mgr.14180) 1055 : cluster [DBG] pgmap v767: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:30 smithi193 bash[28559]: cluster 2023-11-17T07:12:29.135222+0000 mgr.smithi097.swvonl (mgr.14180) 1056 : cluster [DBG] pgmap v768: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:32 smithi193 bash[28559]: cluster 2023-11-17T07:12:31.136558+0000 mgr.smithi097.swvonl (mgr.14180) 1057 : cluster [DBG] pgmap v769: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:34 smithi193 bash[28559]: cluster 2023-11-17T07:12:33.137539+0000 mgr.smithi097.swvonl (mgr.14180) 1058 : cluster [DBG] pgmap v770: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:36 smithi193 bash[28559]: cluster 2023-11-17T07:12:35.138371+0000 mgr.smithi097.swvonl (mgr.14180) 1059 : cluster [DBG] pgmap v771: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:38 smithi193 bash[28559]: cluster 2023-11-17T07:12:37.139326+0000 mgr.smithi097.swvonl (mgr.14180) 1060 : cluster [DBG] pgmap v772: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:40 smithi193 bash[28559]: cluster 2023-11-17T07:12:39.140381+0000 mgr.smithi097.swvonl (mgr.14180) 1061 : cluster [DBG] pgmap v773: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:42 smithi193 bash[28559]: cluster 2023-11-17T07:12:41.141714+0000 mgr.smithi097.swvonl (mgr.14180) 1062 : cluster [DBG] pgmap v774: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:44 smithi193 bash[28559]: cluster 2023-11-17T07:12:43.142640+0000 mgr.smithi097.swvonl (mgr.14180) 1063 : cluster [DBG] pgmap v775: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:46 smithi193 bash[28559]: cluster 2023-11-17T07:12:45.144002+0000 mgr.smithi097.swvonl (mgr.14180) 1064 : cluster [DBG] pgmap v776: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:48 smithi193 bash[28559]: cluster 2023-11-17T07:12:47.144947+0000 mgr.smithi097.swvonl (mgr.14180) 1065 : cluster [DBG] pgmap v777: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:50 smithi193 bash[28559]: cluster 2023-11-17T07:12:49.146006+0000 mgr.smithi097.swvonl (mgr.14180) 1066 : cluster [DBG] pgmap v778: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:52 smithi193 bash[28559]: cluster 2023-11-17T07:12:51.147272+0000 mgr.smithi097.swvonl (mgr.14180) 1067 : cluster [DBG] pgmap v779: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:54 smithi193 bash[28559]: cluster 2023-11-17T07:12:53.148305+0000 mgr.smithi097.swvonl (mgr.14180) 1068 : cluster [DBG] pgmap v780: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:56 smithi193 bash[28559]: cluster 2023-11-17T07:12:55.149744+0000 mgr.smithi097.swvonl (mgr.14180) 1069 : cluster [DBG] pgmap v781: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:12:58 smithi193 bash[28559]: cluster 2023-11-17T07:12:57.150775+0000 mgr.smithi097.swvonl (mgr.14180) 1070 : cluster [DBG] pgmap v782: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.230 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:00 smithi193 bash[28559]: cluster 2023-11-17T07:12:59.151787+0000 mgr.smithi097.swvonl (mgr.14180) 1071 : cluster [DBG] pgmap v783: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:02 smithi193 bash[28559]: cluster 2023-11-17T07:13:01.153082+0000 mgr.smithi097.swvonl (mgr.14180) 1072 : cluster [DBG] pgmap v784: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:04 smithi193 bash[28559]: cluster 2023-11-17T07:13:03.154082+0000 mgr.smithi097.swvonl (mgr.14180) 1073 : cluster [DBG] pgmap v785: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:06 smithi193 bash[28559]: cluster 2023-11-17T07:13:05.155556+0000 mgr.smithi097.swvonl (mgr.14180) 1074 : cluster [DBG] pgmap v786: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:08 smithi193 bash[28559]: cluster 2023-11-17T07:13:07.156605+0000 mgr.smithi097.swvonl (mgr.14180) 1075 : cluster [DBG] pgmap v787: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:10 smithi193 bash[28559]: cluster 2023-11-17T07:13:09.157704+0000 mgr.smithi097.swvonl (mgr.14180) 1076 : cluster [DBG] pgmap v788: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:12 smithi193 bash[28559]: cluster 2023-11-17T07:13:11.159108+0000 mgr.smithi097.swvonl (mgr.14180) 1077 : cluster [DBG] pgmap v789: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:14 smithi193 bash[28559]: cluster 2023-11-17T07:13:13.160099+0000 mgr.smithi097.swvonl (mgr.14180) 1078 : cluster [DBG] pgmap v790: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.131407+0000 mon.smithi097 (mon.0) 942 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.424927+0000 mon.smithi097 (mon.0) 943 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.426585+0000 mon.smithi097 (mon.0) 944 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.434954+0000 mon.smithi097 (mon.0) 945 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.442753+0000 mon.smithi097 (mon.0) 946 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:08.231 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:15 smithi193 bash[28559]: audit 2023-11-17T07:13:15.456345+0000 mon.smithi097 (mon.0) 947 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:16 smithi193 bash[28559]: cluster 2023-11-17T07:13:15.161493+0000 mgr.smithi097.swvonl (mgr.14180) 1079 : cluster [DBG] pgmap v791: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:16 smithi193 bash[28559]: cluster 2023-11-17T07:13:15.435693+0000 mgr.smithi097.swvonl (mgr.14180) 1080 : cluster [DBG] pgmap v792: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:16 smithi193 bash[28559]: cephadm 2023-11-17T07:13:15.462589+0000 mgr.smithi097.swvonl (mgr.14180) 1081 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.tjpsob on smithi193 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:16 smithi193 bash[28559]: cluster 2023-11-17T07:13:15.679467+0000 mon.smithi097 (mon.0) 948 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:16 smithi193 bash[28559]: cluster 2023-11-17T07:13:15.679511+0000 mon.smithi097 (mon.0) 949 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:18 smithi193 bash[28559]: cluster 2023-11-17T07:13:17.436400+0000 mgr.smithi097.swvonl (mgr.14180) 1082 : cluster [DBG] pgmap v793: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.232 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:19 smithi193 bash[28559]: audit 2023-11-17T07:13:19.354467+0000 mon.smithi097 (mon.0) 950 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.tjpsob"}]: dispatch 2023-11-17T07:14:08.251 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: cephadm 2023-11-17T07:13:19.353248+0000 mgr.smithi097.swvonl (mgr.14180) 1083 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:08.252 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.253 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: cephadm 2023-11-17T07:13:19.353757+0000 mgr.smithi097.swvonl (mgr.14180) 1084 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: cephadm 2023-11-17T07:13:19.356392+0000 mgr.smithi097.swvonl (mgr.14180) 1085 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.254 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: cephadm 2023-11-17T07:13:19.361352+0000 mgr.smithi097.swvonl (mgr.14180) 1086 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.eeznnn on smithi097 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:20 smithi193 bash[28559]: cluster 2023-11-17T07:13:19.437260+0000 mgr.smithi097.swvonl (mgr.14180) 1087 : cluster [DBG] pgmap v794: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:22 smithi193 bash[28559]: cluster 2023-11-17T07:13:21.438358+0000 mgr.smithi097.swvonl (mgr.14180) 1088 : cluster [DBG] pgmap v795: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:23 smithi193 bash[28559]: audit 2023-11-17T07:13:23.112951+0000 mon.smithi097 (mon.0) 951 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.eeznnn"}]: dispatch 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cephadm 2023-11-17T07:13:23.111734+0000 mgr.smithi097.swvonl (mgr.14180) 1089 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.255 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.256 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.257 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cephadm 2023-11-17T07:13:23.112271+0000 mgr.smithi097.swvonl (mgr.14180) 1090 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cephadm 2023-11-17T07:13:23.114478+0000 mgr.smithi097.swvonl (mgr.14180) 1091 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cephadm 2023-11-17T07:13:23.117569+0000 mgr.smithi097.swvonl (mgr.14180) 1092 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cephadm 2023-11-17T07:13:23.120335+0000 mgr.smithi097.swvonl (mgr.14180) 1093 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cluster 2023-11-17T07:13:23.121563+0000 mgr.smithi097.swvonl (mgr.14180) 1094 : cluster [DBG] pgmap v796: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: cluster 2023-11-17T07:13:23.715862+0000 mon.smithi097 (mon.0) 952 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:24 smithi193 bash[28559]: audit 2023-11-17T07:13:24.079316+0000 mon.smithi097 (mon.0) 953 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.316 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:26 smithi193 bash[28559]: cluster 2023-11-17T07:13:25.123026+0000 mgr.smithi097.swvonl (mgr.14180) 1095 : cluster [DBG] pgmap v797: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:14:08.317 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:28 smithi193 bash[28559]: cluster 2023-11-17T07:13:27.124129+0000 mgr.smithi097.swvonl (mgr.14180) 1096 : cluster [DBG] pgmap v798: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.317 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:30 smithi193 bash[28559]: cluster 2023-11-17T07:13:29.125220+0000 mgr.smithi097.swvonl (mgr.14180) 1097 : cluster [DBG] pgmap v799: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.317 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:32 smithi193 bash[28559]: cluster 2023-11-17T07:13:31.126600+0000 mgr.smithi097.swvonl (mgr.14180) 1098 : cluster [DBG] pgmap v800: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2023-11-17T07:14:08.317 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:34 smithi193 bash[28559]: cluster 2023-11-17T07:13:33.127485+0000 mgr.smithi097.swvonl (mgr.14180) 1099 : cluster [DBG] pgmap v801: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.318 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:36 smithi193 bash[28559]: cluster 2023-11-17T07:13:35.128875+0000 mgr.smithi097.swvonl (mgr.14180) 1100 : cluster [DBG] pgmap v802: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.318 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:38 smithi193 bash[28559]: cluster 2023-11-17T07:13:37.129858+0000 mgr.smithi097.swvonl (mgr.14180) 1101 : cluster [DBG] pgmap v803: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.318 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:40 smithi193 bash[28559]: cluster 2023-11-17T07:13:39.131476+0000 mgr.smithi097.swvonl (mgr.14180) 1102 : cluster [DBG] pgmap v804: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.319 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:42 smithi193 bash[28559]: cluster 2023-11-17T07:13:41.132790+0000 mgr.smithi097.swvonl (mgr.14180) 1103 : cluster [DBG] pgmap v805: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.319 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:44 smithi193 bash[28559]: cluster 2023-11-17T07:13:43.133749+0000 mgr.smithi097.swvonl (mgr.14180) 1104 : cluster [DBG] pgmap v806: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.319 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:46 smithi193 bash[28559]: cluster 2023-11-17T07:13:45.135138+0000 mgr.smithi097.swvonl (mgr.14180) 1105 : cluster [DBG] pgmap v807: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.320 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:48 smithi193 bash[28559]: cluster 2023-11-17T07:13:47.135845+0000 mgr.smithi097.swvonl (mgr.14180) 1106 : cluster [DBG] pgmap v808: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.320 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:50 smithi193 bash[28559]: cluster 2023-11-17T07:13:49.136869+0000 mgr.smithi097.swvonl (mgr.14180) 1107 : cluster [DBG] pgmap v809: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.320 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:52 smithi193 bash[28559]: cluster 2023-11-17T07:13:51.138311+0000 mgr.smithi097.swvonl (mgr.14180) 1108 : cluster [DBG] pgmap v810: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.320 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:54 smithi193 bash[28559]: cluster 2023-11-17T07:13:53.139422+0000 mgr.smithi097.swvonl (mgr.14180) 1109 : cluster [DBG] pgmap v811: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.320 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:56 smithi193 bash[28559]: cluster 2023-11-17T07:13:55.140390+0000 mgr.smithi097.swvonl (mgr.14180) 1110 : cluster [DBG] pgmap v812: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:13:58 smithi193 bash[28559]: cluster 2023-11-17T07:13:57.141540+0000 mgr.smithi097.swvonl (mgr.14180) 1111 : cluster [DBG] pgmap v813: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:00 smithi193 bash[28559]: cluster 2023-11-17T07:13:59.142784+0000 mgr.smithi097.swvonl (mgr.14180) 1112 : cluster [DBG] pgmap v814: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:02 smithi193 bash[28559]: cluster 2023-11-17T07:14:01.144185+0000 mgr.smithi097.swvonl (mgr.14180) 1113 : cluster [DBG] pgmap v815: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:04 smithi193 bash[28559]: cluster 2023-11-17T07:14:03.145215+0000 mgr.smithi097.swvonl (mgr.14180) 1114 : cluster [DBG] pgmap v816: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:06 smithi193 bash[28559]: cluster 2023-11-17T07:14:05.146676+0000 mgr.smithi097.swvonl (mgr.14180) 1115 : cluster [DBG] pgmap v817: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.323 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:01 smithi097 bash[24958]: audit 2023-11-17T07:12:00.102828+0000 mgr.smithi097.swvonl (mgr.14180) 1030 : audit [DBG] from='client.14960 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:08.323 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:02 smithi097 bash[24958]: cluster 2023-11-17T07:12:01.093226+0000 mgr.smithi097.swvonl (mgr.14180) 1031 : cluster [DBG] pgmap v753: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:04 smithi097 bash[24958]: cluster 2023-11-17T07:12:03.094201+0000 mgr.smithi097.swvonl (mgr.14180) 1032 : cluster [DBG] pgmap v754: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:06 smithi097 bash[24958]: cluster 2023-11-17T07:12:05.095649+0000 mgr.smithi097.swvonl (mgr.14180) 1033 : cluster [DBG] pgmap v755: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.064794+0000 mon.smithi097 (mon.0) 928 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:08.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.363558+0000 mon.smithi097 (mon.0) 929 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:08.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.364882+0000 mon.smithi097 (mon.0) 930 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:08.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.372787+0000 mon.smithi097 (mon.0) 931 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.380377+0000 mon.smithi097 (mon.0) 932 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:08.326 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:07 smithi097 bash[24958]: audit 2023-11-17T07:12:07.392122+0000 mon.smithi097 (mon.0) 933 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.326 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:08 smithi097 bash[24958]: cluster 2023-11-17T07:12:07.096493+0000 mgr.smithi097.swvonl (mgr.14180) 1034 : cluster [DBG] pgmap v756: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.326 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:08 smithi097 bash[24958]: cluster 2023-11-17T07:12:07.373581+0000 mgr.smithi097.swvonl (mgr.14180) 1035 : cluster [DBG] pgmap v757: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.326 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:08 smithi097 bash[24958]: cephadm 2023-11-17T07:12:07.396243+0000 mgr.smithi097.swvonl (mgr.14180) 1036 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.bbwtlw on smithi193 2023-11-17T07:14:08.326 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:08 smithi097 bash[24958]: cluster 2023-11-17T07:12:07.631336+0000 mon.smithi097 (mon.0) 934 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:08 smithi097 bash[24958]: cluster 2023-11-17T07:12:07.631405+0000 mon.smithi097 (mon.0) 935 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:10 smithi097 bash[24958]: cluster 2023-11-17T07:12:09.374428+0000 mgr.smithi097.swvonl (mgr.14180) 1037 : cluster [DBG] pgmap v758: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:11 smithi097 bash[24958]: audit 2023-11-17T07:12:11.357754+0000 mon.smithi097 (mon.0) 936 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.bbwtlw"}]: dispatch 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:11 smithi097 bash[24958]: audit 2023-11-17T07:12:11.537495+0000 mon.smithi097 (mon.0) 937 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: cephadm 2023-11-17T07:12:11.356379+0000 mgr.smithi097.swvonl (mgr.14180) 1038 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.327 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.328 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.329 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: cephadm 2023-11-17T07:12:11.356927+0000 mgr.smithi097.swvonl (mgr.14180) 1039 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: cephadm 2023-11-17T07:12:11.359673+0000 mgr.smithi097.swvonl (mgr.14180) 1040 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.bbwtlw on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bbwtlw 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bbwtlw 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bbwtlw ... 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.330 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: cephadm 2023-11-17T07:12:11.365150+0000 mgr.smithi097.swvonl (mgr.14180) 1041 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.vvyuqu on smithi097 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:12 smithi097 bash[24958]: cluster 2023-11-17T07:12:11.375470+0000 mgr.smithi097.swvonl (mgr.14180) 1042 : cluster [DBG] pgmap v759: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:14 smithi097 bash[24958]: cluster 2023-11-17T07:12:13.376273+0000 mgr.smithi097.swvonl (mgr.14180) 1043 : cluster [DBG] pgmap v760: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:15 smithi097 bash[24958]: audit 2023-11-17T07:12:15.119366+0000 mon.smithi097 (mon.0) 938 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.vvyuqu"}]: dispatch 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cephadm 2023-11-17T07:12:15.118269+0000 mgr.smithi097.swvonl (mgr.14180) 1044 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.331 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.332 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cephadm 2023-11-17T07:12:15.118808+0000 mgr.smithi097.swvonl (mgr.14180) 1045 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cephadm 2023-11-17T07:12:15.120694+0000 mgr.smithi097.swvonl (mgr.14180) 1046 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.vvyuqu on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.333 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-vvyuqu 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.vvyuqu 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.vvyuqu ... 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cephadm 2023-11-17T07:12:15.123749+0000 mgr.smithi097.swvonl (mgr.14180) 1047 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.qqpouv on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cephadm 2023-11-17T07:12:15.126683+0000 mgr.smithi097.swvonl (mgr.14180) 1048 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.tqczco on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.334 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cluster 2023-11-17T07:12:15.128106+0000 mgr.smithi097.swvonl (mgr.14180) 1049 : cluster [DBG] pgmap v761: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:16 smithi097 bash[24958]: cluster 2023-11-17T07:12:15.728482+0000 mon.smithi097 (mon.0) 939 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:18 smithi097 bash[24958]: cluster 2023-11-17T07:12:17.129188+0000 mgr.smithi097.swvonl (mgr.14180) 1050 : cluster [DBG] pgmap v762: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:20 smithi097 bash[24958]: audit 2023-11-17T07:12:19.028544+0000 mon.smithi097 (mon.0) 940 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:20 smithi097 bash[24958]: cluster 2023-11-17T07:12:19.129855+0000 mgr.smithi097.swvonl (mgr.14180) 1051 : cluster [DBG] pgmap v763: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:22 smithi097 bash[24958]: cluster 2023-11-17T07:12:21.131152+0000 mgr.smithi097.swvonl (mgr.14180) 1052 : cluster [DBG] pgmap v764: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:24 smithi097 bash[24958]: cluster 2023-11-17T07:12:23.132049+0000 mgr.smithi097.swvonl (mgr.14180) 1053 : cluster [DBG] pgmap v765: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:26 smithi097 bash[24958]: cluster 2023-11-17T07:12:25.133437+0000 mgr.smithi097.swvonl (mgr.14180) 1054 : cluster [DBG] pgmap v766: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:27 smithi097 bash[24958]: audit 2023-11-17T07:12:26.535949+0000 mon.smithi097 (mon.0) 941 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:28 smithi097 bash[24958]: cluster 2023-11-17T07:12:27.134130+0000 mgr.smithi097.swvonl (mgr.14180) 1055 : cluster [DBG] pgmap v767: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:30 smithi097 bash[24958]: cluster 2023-11-17T07:12:29.135222+0000 mgr.smithi097.swvonl (mgr.14180) 1056 : cluster [DBG] pgmap v768: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:32 smithi097 bash[24958]: cluster 2023-11-17T07:12:31.136558+0000 mgr.smithi097.swvonl (mgr.14180) 1057 : cluster [DBG] pgmap v769: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:34 smithi097 bash[24958]: cluster 2023-11-17T07:12:33.137539+0000 mgr.smithi097.swvonl (mgr.14180) 1058 : cluster [DBG] pgmap v770: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:36 smithi097 bash[24958]: cluster 2023-11-17T07:12:35.138371+0000 mgr.smithi097.swvonl (mgr.14180) 1059 : cluster [DBG] pgmap v771: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:38 smithi097 bash[24958]: cluster 2023-11-17T07:12:37.139326+0000 mgr.smithi097.swvonl (mgr.14180) 1060 : cluster [DBG] pgmap v772: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:40 smithi097 bash[24958]: cluster 2023-11-17T07:12:39.140381+0000 mgr.smithi097.swvonl (mgr.14180) 1061 : cluster [DBG] pgmap v773: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:42 smithi097 bash[24958]: cluster 2023-11-17T07:12:41.141714+0000 mgr.smithi097.swvonl (mgr.14180) 1062 : cluster [DBG] pgmap v774: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:44 smithi097 bash[24958]: cluster 2023-11-17T07:12:43.142640+0000 mgr.smithi097.swvonl (mgr.14180) 1063 : cluster [DBG] pgmap v775: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:46 smithi097 bash[24958]: cluster 2023-11-17T07:12:45.144002+0000 mgr.smithi097.swvonl (mgr.14180) 1064 : cluster [DBG] pgmap v776: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:48 smithi097 bash[24958]: cluster 2023-11-17T07:12:47.144947+0000 mgr.smithi097.swvonl (mgr.14180) 1065 : cluster [DBG] pgmap v777: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:50 smithi097 bash[24958]: cluster 2023-11-17T07:12:49.146006+0000 mgr.smithi097.swvonl (mgr.14180) 1066 : cluster [DBG] pgmap v778: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:52 smithi097 bash[24958]: cluster 2023-11-17T07:12:51.147272+0000 mgr.smithi097.swvonl (mgr.14180) 1067 : cluster [DBG] pgmap v779: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:54 smithi097 bash[24958]: cluster 2023-11-17T07:12:53.148305+0000 mgr.smithi097.swvonl (mgr.14180) 1068 : cluster [DBG] pgmap v780: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:56 smithi097 bash[24958]: cluster 2023-11-17T07:12:55.149744+0000 mgr.smithi097.swvonl (mgr.14180) 1069 : cluster [DBG] pgmap v781: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:12:58 smithi097 bash[24958]: cluster 2023-11-17T07:12:57.150775+0000 mgr.smithi097.swvonl (mgr.14180) 1070 : cluster [DBG] pgmap v782: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:00 smithi097 bash[24958]: cluster 2023-11-17T07:12:59.151787+0000 mgr.smithi097.swvonl (mgr.14180) 1071 : cluster [DBG] pgmap v783: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:02 smithi097 bash[24958]: cluster 2023-11-17T07:13:01.153082+0000 mgr.smithi097.swvonl (mgr.14180) 1072 : cluster [DBG] pgmap v784: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:04 smithi097 bash[24958]: cluster 2023-11-17T07:13:03.154082+0000 mgr.smithi097.swvonl (mgr.14180) 1073 : cluster [DBG] pgmap v785: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:06 smithi097 bash[24958]: cluster 2023-11-17T07:13:05.155556+0000 mgr.smithi097.swvonl (mgr.14180) 1074 : cluster [DBG] pgmap v786: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:08 smithi097 bash[24958]: cluster 2023-11-17T07:13:07.156605+0000 mgr.smithi097.swvonl (mgr.14180) 1075 : cluster [DBG] pgmap v787: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.337 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:10 smithi097 bash[24958]: cluster 2023-11-17T07:13:09.157704+0000 mgr.smithi097.swvonl (mgr.14180) 1076 : cluster [DBG] pgmap v788: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:12 smithi097 bash[24958]: cluster 2023-11-17T07:13:11.159108+0000 mgr.smithi097.swvonl (mgr.14180) 1077 : cluster [DBG] pgmap v789: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:14 smithi097 bash[24958]: cluster 2023-11-17T07:13:13.160099+0000 mgr.smithi097.swvonl (mgr.14180) 1078 : cluster [DBG] pgmap v790: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.131407+0000 mon.smithi097 (mon.0) 942 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.424927+0000 mon.smithi097 (mon.0) 943 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.426585+0000 mon.smithi097 (mon.0) 944 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.434954+0000 mon.smithi097 (mon.0) 945 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.442753+0000 mon.smithi097 (mon.0) 946 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:15 smithi097 bash[24958]: audit 2023-11-17T07:13:15.456345+0000 mon.smithi097 (mon.0) 947 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:16 smithi097 bash[24958]: cluster 2023-11-17T07:13:15.161493+0000 mgr.smithi097.swvonl (mgr.14180) 1079 : cluster [DBG] pgmap v791: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:16 smithi097 bash[24958]: cluster 2023-11-17T07:13:15.435693+0000 mgr.smithi097.swvonl (mgr.14180) 1080 : cluster [DBG] pgmap v792: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:16 smithi097 bash[24958]: cephadm 2023-11-17T07:13:15.462589+0000 mgr.smithi097.swvonl (mgr.14180) 1081 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.tjpsob on smithi193 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:16 smithi097 bash[24958]: cluster 2023-11-17T07:13:15.679467+0000 mon.smithi097 (mon.0) 948 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:08.338 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:16 smithi097 bash[24958]: cluster 2023-11-17T07:13:15.679511+0000 mon.smithi097 (mon.0) 949 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:18 smithi097 bash[24958]: cluster 2023-11-17T07:13:17.436400+0000 mgr.smithi097.swvonl (mgr.14180) 1082 : cluster [DBG] pgmap v793: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:19 smithi097 bash[24958]: audit 2023-11-17T07:13:19.354467+0000 mon.smithi097 (mon.0) 950 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.tjpsob"}]: dispatch 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: cephadm 2023-11-17T07:13:19.353248+0000 mgr.smithi097.swvonl (mgr.14180) 1083 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.339 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.364 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: cephadm 2023-11-17T07:13:19.353757+0000 mgr.smithi097.swvonl (mgr.14180) 1084 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: cephadm 2023-11-17T07:13:19.356392+0000 mgr.smithi097.swvonl (mgr.14180) 1085 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.365 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tjpsob ... 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: cephadm 2023-11-17T07:13:19.361352+0000 mgr.smithi097.swvonl (mgr.14180) 1086 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.eeznnn on smithi097 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:20 smithi097 bash[24958]: cluster 2023-11-17T07:13:19.437260+0000 mgr.smithi097.swvonl (mgr.14180) 1087 : cluster [DBG] pgmap v794: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:22 smithi097 bash[24958]: cluster 2023-11-17T07:13:21.438358+0000 mgr.smithi097.swvonl (mgr.14180) 1088 : cluster [DBG] pgmap v795: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:23 smithi097 bash[24958]: audit 2023-11-17T07:13:23.112951+0000 mon.smithi097 (mon.0) 951 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.eeznnn"}]: dispatch 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cephadm 2023-11-17T07:13:23.111734+0000 mgr.smithi097.swvonl (mgr.14180) 1089 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.366 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:08.367 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.368 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cephadm 2023-11-17T07:13:23.112271+0000 mgr.smithi097.swvonl (mgr.14180) 1090 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cephadm 2023-11-17T07:13:23.114478+0000 mgr.smithi097.swvonl (mgr.14180) 1091 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.eeznnn ... 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:08.369 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cephadm 2023-11-17T07:13:23.117569+0000 mgr.smithi097.swvonl (mgr.14180) 1092 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cephadm 2023-11-17T07:13:23.120335+0000 mgr.smithi097.swvonl (mgr.14180) 1093 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cluster 2023-11-17T07:13:23.121563+0000 mgr.smithi097.swvonl (mgr.14180) 1094 : cluster [DBG] pgmap v796: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: cluster 2023-11-17T07:13:23.715862+0000 mon.smithi097 (mon.0) 952 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:24 smithi097 bash[24958]: audit 2023-11-17T07:13:24.079316+0000 mon.smithi097 (mon.0) 953 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:26 smithi097 bash[24958]: cluster 2023-11-17T07:13:25.123026+0000 mgr.smithi097.swvonl (mgr.14180) 1095 : cluster [DBG] pgmap v797: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 205 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:28 smithi097 bash[24958]: cluster 2023-11-17T07:13:27.124129+0000 mgr.smithi097.swvonl (mgr.14180) 1096 : cluster [DBG] pgmap v798: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:30 smithi097 bash[24958]: cluster 2023-11-17T07:13:29.125220+0000 mgr.smithi097.swvonl (mgr.14180) 1097 : cluster [DBG] pgmap v799: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:32 smithi097 bash[24958]: cluster 2023-11-17T07:13:31.126600+0000 mgr.smithi097.swvonl (mgr.14180) 1098 : cluster [DBG] pgmap v800: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:34 smithi097 bash[24958]: cluster 2023-11-17T07:13:33.127485+0000 mgr.smithi097.swvonl (mgr.14180) 1099 : cluster [DBG] pgmap v801: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:36 smithi097 bash[24958]: cluster 2023-11-17T07:13:35.128875+0000 mgr.smithi097.swvonl (mgr.14180) 1100 : cluster [DBG] pgmap v802: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.370 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:38 smithi097 bash[24958]: cluster 2023-11-17T07:13:37.129858+0000 mgr.smithi097.swvonl (mgr.14180) 1101 : cluster [DBG] pgmap v803: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:40 smithi097 bash[24958]: cluster 2023-11-17T07:13:39.131476+0000 mgr.smithi097.swvonl (mgr.14180) 1102 : cluster [DBG] pgmap v804: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:42 smithi097 bash[24958]: cluster 2023-11-17T07:13:41.132790+0000 mgr.smithi097.swvonl (mgr.14180) 1103 : cluster [DBG] pgmap v805: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:44 smithi097 bash[24958]: cluster 2023-11-17T07:13:43.133749+0000 mgr.smithi097.swvonl (mgr.14180) 1104 : cluster [DBG] pgmap v806: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:46 smithi097 bash[24958]: cluster 2023-11-17T07:13:45.135138+0000 mgr.smithi097.swvonl (mgr.14180) 1105 : cluster [DBG] pgmap v807: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:48 smithi097 bash[24958]: cluster 2023-11-17T07:13:47.135845+0000 mgr.smithi097.swvonl (mgr.14180) 1106 : cluster [DBG] pgmap v808: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:50 smithi097 bash[24958]: cluster 2023-11-17T07:13:49.136869+0000 mgr.smithi097.swvonl (mgr.14180) 1107 : cluster [DBG] pgmap v809: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:52 smithi097 bash[24958]: cluster 2023-11-17T07:13:51.138311+0000 mgr.smithi097.swvonl (mgr.14180) 1108 : cluster [DBG] pgmap v810: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:54 smithi097 bash[24958]: cluster 2023-11-17T07:13:53.139422+0000 mgr.smithi097.swvonl (mgr.14180) 1109 : cluster [DBG] pgmap v811: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:56 smithi097 bash[24958]: cluster 2023-11-17T07:13:55.140390+0000 mgr.smithi097.swvonl (mgr.14180) 1110 : cluster [DBG] pgmap v812: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:13:58 smithi097 bash[24958]: cluster 2023-11-17T07:13:57.141540+0000 mgr.smithi097.swvonl (mgr.14180) 1111 : cluster [DBG] pgmap v813: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:00 smithi097 bash[24958]: cluster 2023-11-17T07:13:59.142784+0000 mgr.smithi097.swvonl (mgr.14180) 1112 : cluster [DBG] pgmap v814: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:02 smithi097 bash[24958]: cluster 2023-11-17T07:14:01.144185+0000 mgr.smithi097.swvonl (mgr.14180) 1113 : cluster [DBG] pgmap v815: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:04 smithi097 bash[24958]: cluster 2023-11-17T07:14:03.145215+0000 mgr.smithi097.swvonl (mgr.14180) 1114 : cluster [DBG] pgmap v816: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:08.371 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:06 smithi097 bash[24958]: cluster 2023-11-17T07:14:05.146676+0000 mgr.smithi097.swvonl (mgr.14180) 1115 : cluster [DBG] pgmap v817: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:09.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:09.253 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:08 smithi097 bash[24958]: cluster 2023-11-17T07:14:07.147740+0000 mgr.smithi097.swvonl (mgr.14180) 1116 : cluster [DBG] pgmap v818: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:09.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:08 smithi193 bash[28559]: cluster 2023-11-17T07:14:07.147740+0000 mgr.smithi097.swvonl (mgr.14180) 1116 : cluster [DBG] pgmap v818: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:10 smithi097 bash[24958]: cluster 2023-11-17T07:14:09.148794+0000 mgr.smithi097.swvonl (mgr.14180) 1117 : cluster [DBG] pgmap v819: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:11.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:10 smithi193 bash[28559]: cluster 2023-11-17T07:14:09.148794+0000 mgr.smithi097.swvonl (mgr.14180) 1117 : cluster [DBG] pgmap v819: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:12.826 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:12.826 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:12:15.126526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.tqczco on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:19.356110Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\nDeploy daemon haproxy.nfs.foo.smithi193.tjpsob ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.114184Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\nDeploy daemon haproxy.nfs.foo.smithi097.eeznnn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.117423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:13.118 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:12 smithi097 bash[24958]: cluster 2023-11-17T07:14:11.150094+0000 mgr.smithi097.swvonl (mgr.14180) 1118 : cluster [DBG] pgmap v820: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:12 smithi193 bash[28559]: cluster 2023-11-17T07:14:11.150094+0000 mgr.smithi097.swvonl (mgr.14180) 1118 : cluster [DBG] pgmap v820: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:13.746 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:14.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:13 smithi097 bash[24958]: audit 2023-11-17T07:14:12.812441+0000 mgr.smithi097.swvonl (mgr.14180) 1119 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:14.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:13 smithi193 bash[28559]: audit 2023-11-17T07:14:12.812441+0000 mgr.smithi097.swvonl (mgr.14180) 1119 : audit [DBG] from='client.14964 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:14.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:15.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:14 smithi097 bash[24958]: cluster 2023-11-17T07:14:13.150836+0000 mgr.smithi097.swvonl (mgr.14180) 1120 : cluster [DBG] pgmap v821: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:15.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:14 smithi193 bash[28559]: cluster 2023-11-17T07:14:13.150836+0000 mgr.smithi097.swvonl (mgr.14180) 1120 : cluster [DBG] pgmap v821: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:17.189 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:16 smithi097 bash[24958]: cluster 2023-11-17T07:14:15.152122+0000 mgr.smithi097.swvonl (mgr.14180) 1121 : cluster [DBG] pgmap v822: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:17.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:16 smithi193 bash[28559]: cluster 2023-11-17T07:14:15.152122+0000 mgr.smithi097.swvonl (mgr.14180) 1121 : cluster [DBG] pgmap v822: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:18.612 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:12:15.126526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.tqczco on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:19.356110Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\nDeploy daemon haproxy.nfs.foo.smithi193.tjpsob ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.114184Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\nDeploy daemon haproxy.nfs.foo.smithi097.eeznnn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.117423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:19.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:18 smithi097 bash[24958]: cluster 2023-11-17T07:14:17.153154+0000 mgr.smithi097.swvonl (mgr.14180) 1122 : cluster [DBG] pgmap v823: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:19.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:18 smithi193 bash[28559]: cluster 2023-11-17T07:14:17.153154+0000 mgr.smithi097.swvonl (mgr.14180) 1122 : cluster [DBG] pgmap v823: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:19.485 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:20.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:19 smithi097 bash[24958]: audit 2023-11-17T07:14:18.601285+0000 mgr.smithi097.swvonl (mgr.14180) 1123 : audit [DBG] from='client.14968 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:19 smithi193 bash[28559]: audit 2023-11-17T07:14:18.601285+0000 mgr.smithi097.swvonl (mgr.14180) 1123 : audit [DBG] from='client.14968 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:20.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:20 smithi097 bash[24958]: cluster 2023-11-17T07:14:19.153811+0000 mgr.smithi097.swvonl (mgr.14180) 1124 : cluster [DBG] pgmap v824: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:21.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:20 smithi193 bash[28559]: cluster 2023-11-17T07:14:19.153811+0000 mgr.smithi097.swvonl (mgr.14180) 1124 : cluster [DBG] pgmap v824: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:23.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:22 smithi097 bash[24958]: cluster 2023-11-17T07:14:21.155144+0000 mgr.smithi097.swvonl (mgr.14180) 1125 : cluster [DBG] pgmap v825: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:23.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:22 smithi193 bash[28559]: cluster 2023-11-17T07:14:21.155144+0000 mgr.smithi097.swvonl (mgr.14180) 1125 : cluster [DBG] pgmap v825: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:24.269 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:24.269 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:12:15.126526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.tqczco on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:19.356110Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\nDeploy daemon haproxy.nfs.foo.smithi193.tjpsob ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.114184Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\nDeploy daemon haproxy.nfs.foo.smithi097.eeznnn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.117423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.125401+0000 mon.smithi097 (mon.0) 954 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.434707+0000 mon.smithi097 (mon.0) 955 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.436143+0000 mon.smithi097 (mon.0) 956 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.443920+0000 mon.smithi097 (mon.0) 957 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.451495+0000 mon.smithi097 (mon.0) 958 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:23 smithi097 bash[24958]: audit 2023-11-17T07:14:23.465101+0000 mon.smithi097 (mon.0) 959 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.125401+0000 mon.smithi097 (mon.0) 954 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:14:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.434707+0000 mon.smithi097 (mon.0) 955 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:14:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.436143+0000 mon.smithi097 (mon.0) 956 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:14:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.443920+0000 mon.smithi097 (mon.0) 957 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.451495+0000 mon.smithi097 (mon.0) 958 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:14:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:23 smithi193 bash[28559]: audit 2023-11-17T07:14:23.465101+0000 mon.smithi097 (mon.0) 959 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:25.098 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:25.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:24 smithi193 bash[28559]: cluster 2023-11-17T07:14:23.156014+0000 mgr.smithi097.swvonl (mgr.14180) 1126 : cluster [DBG] pgmap v826: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:25.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:24 smithi193 bash[28559]: cluster 2023-11-17T07:14:23.444826+0000 mgr.smithi097.swvonl (mgr.14180) 1127 : cluster [DBG] pgmap v827: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:25.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:24 smithi193 bash[28559]: cephadm 2023-11-17T07:14:23.468264+0000 mgr.smithi097.swvonl (mgr.14180) 1128 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jjsbhi on smithi193 2023-11-17T07:14:25.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:24 smithi193 bash[28559]: cluster 2023-11-17T07:14:23.905946+0000 mon.smithi097 (mon.0) 960 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:25.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:24 smithi193 bash[28559]: cluster 2023-11-17T07:14:23.906009+0000 mon.smithi097 (mon.0) 961 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:25.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:24 smithi097 bash[24958]: cluster 2023-11-17T07:14:23.156014+0000 mgr.smithi097.swvonl (mgr.14180) 1126 : cluster [DBG] pgmap v826: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:25.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:24 smithi097 bash[24958]: cluster 2023-11-17T07:14:23.444826+0000 mgr.smithi097.swvonl (mgr.14180) 1127 : cluster [DBG] pgmap v827: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:25.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:24 smithi097 bash[24958]: cephadm 2023-11-17T07:14:23.468264+0000 mgr.smithi097.swvonl (mgr.14180) 1128 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jjsbhi on smithi193 2023-11-17T07:14:25.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:24 smithi097 bash[24958]: cluster 2023-11-17T07:14:23.905946+0000 mon.smithi097 (mon.0) 960 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:14:25.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:24 smithi097 bash[24958]: cluster 2023-11-17T07:14:23.906009+0000 mon.smithi097 (mon.0) 961 : cluster [INF] Cluster is now healthy 2023-11-17T07:14:26.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:26.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:25 smithi097 bash[24958]: audit 2023-11-17T07:14:24.262110+0000 mgr.smithi097.swvonl (mgr.14180) 1129 : audit [DBG] from='client.14972 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:26.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:25 smithi193 bash[28559]: audit 2023-11-17T07:14:24.262110+0000 mgr.smithi097.swvonl (mgr.14180) 1129 : audit [DBG] from='client.14972 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:27.205 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:26 smithi193 bash[28559]: cluster 2023-11-17T07:14:25.445938+0000 mgr.smithi097.swvonl (mgr.14180) 1130 : cluster [DBG] pgmap v828: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:27.205 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:26 smithi193 bash[28559]: audit 2023-11-17T07:14:26.543757+0000 mon.smithi097 (mon.0) 962 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:27.277 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:26 smithi097 bash[24958]: cluster 2023-11-17T07:14:25.445938+0000 mgr.smithi097.swvonl (mgr.14180) 1130 : cluster [DBG] pgmap v828: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:27.278 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:26 smithi097 bash[24958]: audit 2023-11-17T07:14:26.543757+0000 mon.smithi097 (mon.0) 962 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: cephadm 2023-11-17T07:14:27.275983+0000 mgr.smithi097.swvonl (mgr.14180) 1131 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:28.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: cephadm 2023-11-17T07:14:27.276503+0000 mgr.smithi097.swvonl (mgr.14180) 1132 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: audit 2023-11-17T07:14:27.277138+0000 mon.smithi097 (mon.0) 963 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jjsbhi"}]: dispatch 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: cephadm 2023-11-17T07:14:27.278658+0000 mgr.smithi097.swvonl (mgr.14180) 1133 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: cephadm 2023-11-17T07:14:27.283387+0000 mgr.smithi097.swvonl (mgr.14180) 1134 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.wwvghf on smithi097 2023-11-17T07:14:28.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:27 smithi097 bash[24958]: cluster 2023-11-17T07:14:27.446551+0000 mgr.smithi097.swvonl (mgr.14180) 1135 : cluster [DBG] pgmap v829: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:28.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: cephadm 2023-11-17T07:14:27.275983+0000 mgr.smithi097.swvonl (mgr.14180) 1131 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: cephadm 2023-11-17T07:14:27.276503+0000 mgr.smithi097.swvonl (mgr.14180) 1132 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: audit 2023-11-17T07:14:27.277138+0000 mon.smithi097 (mon.0) 963 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jjsbhi"}]: dispatch 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: cephadm 2023-11-17T07:14:27.278658+0000 mgr.smithi097.swvonl (mgr.14180) 1133 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jjsbhi ... 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:28.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:28.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:28.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:28.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: cephadm 2023-11-17T07:14:27.283387+0000 mgr.smithi097.swvonl (mgr.14180) 1134 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.wwvghf on smithi097 2023-11-17T07:14:28.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:27 smithi193 bash[28559]: cluster 2023-11-17T07:14:27.446551+0000 mgr.smithi097.swvonl (mgr.14180) 1135 : cluster [DBG] pgmap v829: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:29.855 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:29.855 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:19.356110Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.tjpsob on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tjpsob\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tjpsob\nDeploy daemon haproxy.nfs.foo.smithi193.tjpsob ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.114184Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.eeznnn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-eeznnn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.eeznnn\nDeploy daemon haproxy.nfs.foo.smithi097.eeznnn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:13:23.117423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rpyaci on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:30.682 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:30 smithi097 bash[24958]: cluster 2023-11-17T07:14:29.447378+0000 mgr.smithi097.swvonl (mgr.14180) 1136 : cluster [DBG] pgmap v830: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:30.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:30 smithi097 bash[24958]: audit 2023-11-17T07:14:29.847346+0000 mgr.smithi097.swvonl (mgr.14180) 1137 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:30 smithi193 bash[28559]: cluster 2023-11-17T07:14:29.447378+0000 mgr.smithi097.swvonl (mgr.14180) 1136 : cluster [DBG] pgmap v830: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:14:30.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:30 smithi193 bash[28559]: audit 2023-11-17T07:14:29.847346+0000 mgr.smithi097.swvonl (mgr.14180) 1137 : audit [DBG] from='client.14976 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:31.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:31.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:31 smithi097 bash[24958]: audit 2023-11-17T07:14:31.016720+0000 mon.smithi097 (mon.0) 964 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.wwvghf"}]: dispatch 2023-11-17T07:14:31.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:31 smithi193 bash[28559]: audit 2023-11-17T07:14:31.016720+0000 mon.smithi097 (mon.0) 964 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.wwvghf"}]: dispatch 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cephadm 2023-11-17T07:14:31.015550+0000 mgr.smithi097.swvonl (mgr.14180) 1138 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cephadm 2023-11-17T07:14:31.016112+0000 mgr.smithi097.swvonl (mgr.14180) 1139 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cephadm 2023-11-17T07:14:31.018407+0000 mgr.smithi097.swvonl (mgr.14180) 1140 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cephadm 2023-11-17T07:14:31.022753+0000 mgr.smithi097.swvonl (mgr.14180) 1141 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cephadm 2023-11-17T07:14:31.024588+0000 mgr.smithi097.swvonl (mgr.14180) 1142 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:32.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cluster 2023-11-17T07:14:31.025794+0000 mgr.smithi097.swvonl (mgr.14180) 1143 : cluster [DBG] pgmap v831: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:14:32.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:32 smithi097 bash[24958]: cluster 2023-11-17T07:14:31.503922+0000 mon.smithi097 (mon.0) 965 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cephadm 2023-11-17T07:14:31.015550+0000 mgr.smithi097.swvonl (mgr.14180) 1138 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cephadm 2023-11-17T07:14:31.016112+0000 mgr.smithi097.swvonl (mgr.14180) 1139 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cephadm 2023-11-17T07:14:31.018407+0000 mgr.smithi097.swvonl (mgr.14180) 1140 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.wwvghf ... 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:14:32.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cephadm 2023-11-17T07:14:31.022753+0000 mgr.smithi097.swvonl (mgr.14180) 1141 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cephadm 2023-11-17T07:14:31.024588+0000 mgr.smithi097.swvonl (mgr.14180) 1142 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cluster 2023-11-17T07:14:31.025794+0000 mgr.smithi097.swvonl (mgr.14180) 1143 : cluster [DBG] pgmap v831: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:14:32.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:32 smithi193 bash[28559]: cluster 2023-11-17T07:14:31.503922+0000 mon.smithi097 (mon.0) 965 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:14:34.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:34 smithi097 bash[24958]: cluster 2023-11-17T07:14:33.026703+0000 mgr.smithi097.swvonl (mgr.14180) 1144 : cluster [DBG] pgmap v832: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:14:34.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:34 smithi097 bash[24958]: audit 2023-11-17T07:14:34.132749+0000 mon.smithi097 (mon.0) 966 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:34.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:34 smithi193 bash[28559]: cluster 2023-11-17T07:14:33.026703+0000 mgr.smithi097.swvonl (mgr.14180) 1144 : cluster [DBG] pgmap v832: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:14:34.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:34 smithi193 bash[28559]: audit 2023-11-17T07:14:34.132749+0000 mon.smithi097 (mon.0) 966 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:35.362 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:35.362 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:36.250 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:36 smithi097 bash[24958]: cluster 2023-11-17T07:14:35.028032+0000 mgr.smithi097.swvonl (mgr.14180) 1145 : cluster [DBG] pgmap v833: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:36.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:36 smithi097 bash[24958]: audit 2023-11-17T07:14:35.350858+0000 mgr.smithi097.swvonl (mgr.14180) 1146 : audit [DBG] from='client.14980 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:36.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:36 smithi193 bash[28559]: cluster 2023-11-17T07:14:35.028032+0000 mgr.smithi097.swvonl (mgr.14180) 1145 : cluster [DBG] pgmap v833: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:36.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:36 smithi193 bash[28559]: audit 2023-11-17T07:14:35.350858+0000 mgr.smithi097.swvonl (mgr.14180) 1146 : audit [DBG] from='client.14980 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:37.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:38.790 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:38 smithi097 bash[24958]: cluster 2023-11-17T07:14:37.029055+0000 mgr.smithi097.swvonl (mgr.14180) 1147 : cluster [DBG] pgmap v834: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:38.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:38 smithi193 bash[28559]: cluster 2023-11-17T07:14:37.029055+0000 mgr.smithi097.swvonl (mgr.14180) 1147 : cluster [DBG] pgmap v834: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:40.788 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:40.788 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:40.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:40 smithi097 bash[24958]: cluster 2023-11-17T07:14:39.030178+0000 mgr.smithi097.swvonl (mgr.14180) 1148 : cluster [DBG] pgmap v835: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:40.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:40 smithi193 bash[28559]: cluster 2023-11-17T07:14:39.030178+0000 mgr.smithi097.swvonl (mgr.14180) 1148 : cluster [DBG] pgmap v835: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:14:41.683 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:41.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:41 smithi193 bash[28559]: audit 2023-11-17T07:14:40.776959+0000 mgr.smithi097.swvonl (mgr.14180) 1149 : audit [DBG] from='client.14984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:41.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:41 smithi193 bash[28559]: audit 2023-11-17T07:14:41.541736+0000 mon.smithi097 (mon.0) 967 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:42.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:41 smithi097 bash[24958]: audit 2023-11-17T07:14:40.776959+0000 mgr.smithi097.swvonl (mgr.14180) 1149 : audit [DBG] from='client.14984 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:42.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:41 smithi097 bash[24958]: audit 2023-11-17T07:14:41.541736+0000 mon.smithi097 (mon.0) 967 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:14:42.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:42.882 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:42 smithi097 bash[24958]: cluster 2023-11-17T07:14:41.031494+0000 mgr.smithi097.swvonl (mgr.14180) 1150 : cluster [DBG] pgmap v836: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:14:43.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:42 smithi193 bash[28559]: cluster 2023-11-17T07:14:41.031494+0000 mgr.smithi097.swvonl (mgr.14180) 1150 : cluster [DBG] pgmap v836: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:14:44.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:44 smithi097 bash[24958]: cluster 2023-11-17T07:14:43.032530+0000 mgr.smithi097.swvonl (mgr.14180) 1151 : cluster [DBG] pgmap v837: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:45.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:44 smithi193 bash[28559]: cluster 2023-11-17T07:14:43.032530+0000 mgr.smithi097.swvonl (mgr.14180) 1151 : cluster [DBG] pgmap v837: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:46.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:46.521 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:46.822 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:46 smithi097 bash[24958]: cluster 2023-11-17T07:14:45.033868+0000 mgr.smithi097.swvonl (mgr.14180) 1152 : cluster [DBG] pgmap v838: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:47.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:46 smithi193 bash[28559]: cluster 2023-11-17T07:14:45.033868+0000 mgr.smithi097.swvonl (mgr.14180) 1152 : cluster [DBG] pgmap v838: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:47.390 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:47.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:47 smithi097 bash[24958]: audit 2023-11-17T07:14:46.509550+0000 mgr.smithi097.swvonl (mgr.14180) 1153 : audit [DBG] from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:48.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:47 smithi193 bash[28559]: audit 2023-11-17T07:14:46.509550+0000 mgr.smithi097.swvonl (mgr.14180) 1153 : audit [DBG] from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:48.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:48.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:48 smithi097 bash[24958]: cluster 2023-11-17T07:14:47.034973+0000 mgr.smithi097.swvonl (mgr.14180) 1154 : cluster [DBG] pgmap v839: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:49.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:48 smithi193 bash[28559]: cluster 2023-11-17T07:14:47.034973+0000 mgr.smithi097.swvonl (mgr.14180) 1154 : cluster [DBG] pgmap v839: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:51.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:50 smithi097 bash[24958]: cluster 2023-11-17T07:14:49.036096+0000 mgr.smithi097.swvonl (mgr.14180) 1155 : cluster [DBG] pgmap v840: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:51.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:50 smithi193 bash[28559]: cluster 2023-11-17T07:14:49.036096+0000 mgr.smithi097.swvonl (mgr.14180) 1155 : cluster [DBG] pgmap v840: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:52.099 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:52.099 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:52.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:52 smithi097 bash[24958]: cluster 2023-11-17T07:14:51.037402+0000 mgr.smithi097.swvonl (mgr.14180) 1156 : cluster [DBG] pgmap v841: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:53.042 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:53.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:52 smithi193 bash[28559]: cluster 2023-11-17T07:14:51.037402+0000 mgr.smithi097.swvonl (mgr.14180) 1156 : cluster [DBG] pgmap v841: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:14:54.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:14:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:53 smithi097 bash[24958]: audit 2023-11-17T07:14:52.092219+0000 mgr.smithi097.swvonl (mgr.14180) 1157 : audit [DBG] from='client.14992 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:54.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:53 smithi193 bash[28559]: audit 2023-11-17T07:14:52.092219+0000 mgr.smithi097.swvonl (mgr.14180) 1157 : audit [DBG] from='client.14992 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:55.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:54 smithi097 bash[24958]: cluster 2023-11-17T07:14:53.038337+0000 mgr.smithi097.swvonl (mgr.14180) 1158 : cluster [DBG] pgmap v842: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:54 smithi193 bash[28559]: cluster 2023-11-17T07:14:53.038337+0000 mgr.smithi097.swvonl (mgr.14180) 1158 : cluster [DBG] pgmap v842: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:56.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:56 smithi097 bash[24958]: cluster 2023-11-17T07:14:55.039218+0000 mgr.smithi097.swvonl (mgr.14180) 1159 : cluster [DBG] pgmap v843: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:57.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:56 smithi193 bash[28559]: cluster 2023-11-17T07:14:55.039218+0000 mgr.smithi097.swvonl (mgr.14180) 1159 : cluster [DBG] pgmap v843: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:57.840 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:14:57.841 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:14:58.669 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:14:59.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:58 smithi097 bash[24958]: cluster 2023-11-17T07:14:57.039810+0000 mgr.smithi097.swvonl (mgr.14180) 1160 : cluster [DBG] pgmap v844: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:59.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:14:58 smithi097 bash[24958]: audit 2023-11-17T07:14:57.829629+0000 mgr.smithi097.swvonl (mgr.14180) 1161 : audit [DBG] from='client.14996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:59.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:58 smithi193 bash[28559]: cluster 2023-11-17T07:14:57.039810+0000 mgr.smithi097.swvonl (mgr.14180) 1160 : cluster [DBG] pgmap v844: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:14:59.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:14:58 smithi193 bash[28559]: audit 2023-11-17T07:14:57.829629+0000 mgr.smithi097.swvonl (mgr.14180) 1161 : audit [DBG] from='client.14996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:14:59.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:00.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:00 smithi097 bash[24958]: cluster 2023-11-17T07:14:59.040888+0000 mgr.smithi097.swvonl (mgr.14180) 1162 : cluster [DBG] pgmap v845: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:01.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:00 smithi193 bash[28559]: cluster 2023-11-17T07:14:59.040888+0000 mgr.smithi097.swvonl (mgr.14180) 1162 : cluster [DBG] pgmap v845: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:02.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:02 smithi097 bash[24958]: cluster 2023-11-17T07:15:01.041935+0000 mgr.smithi097.swvonl (mgr.14180) 1163 : cluster [DBG] pgmap v846: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:03.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:02 smithi193 bash[28559]: cluster 2023-11-17T07:15:01.041935+0000 mgr.smithi097.swvonl (mgr.14180) 1163 : cluster [DBG] pgmap v846: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:03.321 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:03.321 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:04.252 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:04 smithi097 bash[24958]: cluster 2023-11-17T07:15:03.042620+0000 mgr.smithi097.swvonl (mgr.14180) 1164 : cluster [DBG] pgmap v847: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:04 smithi097 bash[24958]: audit 2023-11-17T07:15:03.309089+0000 mgr.smithi097.swvonl (mgr.14180) 1165 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:04 smithi193 bash[28559]: cluster 2023-11-17T07:15:03.042620+0000 mgr.smithi097.swvonl (mgr.14180) 1164 : cluster [DBG] pgmap v847: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:05.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:04 smithi193 bash[28559]: audit 2023-11-17T07:15:03.309089+0000 mgr.smithi097.swvonl (mgr.14180) 1165 : audit [DBG] from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:05.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:07.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:06 smithi097 bash[24958]: cluster 2023-11-17T07:15:05.043957+0000 mgr.smithi097.swvonl (mgr.14180) 1166 : cluster [DBG] pgmap v848: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:07.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:06 smithi193 bash[28559]: cluster 2023-11-17T07:15:05.043957+0000 mgr.smithi097.swvonl (mgr.14180) 1166 : cluster [DBG] pgmap v848: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:08.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:08 smithi097 bash[24958]: cluster 2023-11-17T07:15:07.045142+0000 mgr.smithi097.swvonl (mgr.14180) 1167 : cluster [DBG] pgmap v849: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:08.817 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:08.818 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:09.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:08 smithi193 bash[28559]: cluster 2023-11-17T07:15:07.045142+0000 mgr.smithi097.swvonl (mgr.14180) 1167 : cluster [DBG] pgmap v849: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:09.722 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:10.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:09 smithi097 bash[24958]: audit 2023-11-17T07:15:08.804357+0000 mgr.smithi097.swvonl (mgr.14180) 1168 : audit [DBG] from='client.15004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:10.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:09 smithi193 bash[28559]: audit 2023-11-17T07:15:08.804357+0000 mgr.smithi097.swvonl (mgr.14180) 1168 : audit [DBG] from='client.15004 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:10.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:10.926 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:10 smithi097 bash[24958]: cluster 2023-11-17T07:15:09.046076+0000 mgr.smithi097.swvonl (mgr.14180) 1169 : cluster [DBG] pgmap v850: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:11.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:10 smithi193 bash[28559]: cluster 2023-11-17T07:15:09.046076+0000 mgr.smithi097.swvonl (mgr.14180) 1169 : cluster [DBG] pgmap v850: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:12 smithi097 bash[24958]: cluster 2023-11-17T07:15:11.046939+0000 mgr.smithi097.swvonl (mgr.14180) 1170 : cluster [DBG] pgmap v851: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:13.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:12 smithi193 bash[28559]: cluster 2023-11-17T07:15:11.046939+0000 mgr.smithi097.swvonl (mgr.14180) 1170 : cluster [DBG] pgmap v851: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:14.468 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:14.468 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:14 smithi097 bash[24958]: cluster 2023-11-17T07:15:13.047557+0000 mgr.smithi097.swvonl (mgr.14180) 1171 : cluster [DBG] pgmap v852: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:14 smithi193 bash[28559]: cluster 2023-11-17T07:15:13.047557+0000 mgr.smithi097.swvonl (mgr.14180) 1171 : cluster [DBG] pgmap v852: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:15.322 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:15 smithi097 bash[24958]: audit 2023-11-17T07:15:14.455944+0000 mgr.smithi097.swvonl (mgr.14180) 1172 : audit [DBG] from='client.15008 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:15 smithi193 bash[28559]: audit 2023-11-17T07:15:14.455944+0000 mgr.smithi097.swvonl (mgr.14180) 1172 : audit [DBG] from='client.15008 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:16.323 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:17.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:16 smithi097 bash[24958]: cluster 2023-11-17T07:15:15.048464+0000 mgr.smithi097.swvonl (mgr.14180) 1173 : cluster [DBG] pgmap v853: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:17.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:16 smithi193 bash[28559]: cluster 2023-11-17T07:15:15.048464+0000 mgr.smithi097.swvonl (mgr.14180) 1173 : cluster [DBG] pgmap v853: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:18.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:18 smithi097 bash[24958]: cluster 2023-11-17T07:15:17.049532+0000 mgr.smithi097.swvonl (mgr.14180) 1174 : cluster [DBG] pgmap v854: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:19.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:18 smithi193 bash[28559]: cluster 2023-11-17T07:15:17.049532+0000 mgr.smithi097.swvonl (mgr.14180) 1174 : cluster [DBG] pgmap v854: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:19.921 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:19.922 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:20.800 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:20 smithi097 bash[24958]: cluster 2023-11-17T07:15:19.050686+0000 mgr.smithi097.swvonl (mgr.14180) 1175 : cluster [DBG] pgmap v855: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:20 smithi097 bash[24958]: audit 2023-11-17T07:15:19.909493+0000 mgr.smithi097.swvonl (mgr.14180) 1176 : audit [DBG] from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:20 smithi193 bash[28559]: cluster 2023-11-17T07:15:19.050686+0000 mgr.smithi097.swvonl (mgr.14180) 1175 : cluster [DBG] pgmap v855: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:20 smithi193 bash[28559]: audit 2023-11-17T07:15:19.909493+0000 mgr.smithi097.swvonl (mgr.14180) 1176 : audit [DBG] from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:21.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:22.969 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:22 smithi097 bash[24958]: cluster 2023-11-17T07:15:21.052026+0000 mgr.smithi097.swvonl (mgr.14180) 1177 : cluster [DBG] pgmap v856: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:23.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:22 smithi193 bash[28559]: cluster 2023-11-17T07:15:21.052026+0000 mgr.smithi097.swvonl (mgr.14180) 1177 : cluster [DBG] pgmap v856: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:25.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:24 smithi097 bash[24958]: cluster 2023-11-17T07:15:23.053031+0000 mgr.smithi097.swvonl (mgr.14180) 1178 : cluster [DBG] pgmap v857: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:25.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:24 smithi193 bash[28559]: cluster 2023-11-17T07:15:23.053031+0000 mgr.smithi097.swvonl (mgr.14180) 1178 : cluster [DBG] pgmap v857: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:25.372 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:25.372 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:26.272 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:27.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:26 smithi097 bash[24958]: cluster 2023-11-17T07:15:25.054332+0000 mgr.smithi097.swvonl (mgr.14180) 1179 : cluster [DBG] pgmap v858: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:27.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:26 smithi097 bash[24958]: audit 2023-11-17T07:15:25.361730+0000 mgr.smithi097.swvonl (mgr.14180) 1180 : audit [DBG] from='client.15016 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:27.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:26 smithi193 bash[28559]: cluster 2023-11-17T07:15:25.054332+0000 mgr.smithi097.swvonl (mgr.14180) 1179 : cluster [DBG] pgmap v858: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:27.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:26 smithi193 bash[28559]: audit 2023-11-17T07:15:25.361730+0000 mgr.smithi097.swvonl (mgr.14180) 1180 : audit [DBG] from='client.15016 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:27.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:28.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:28 smithi097 bash[24958]: cluster 2023-11-17T07:15:27.055352+0000 mgr.smithi097.swvonl (mgr.14180) 1181 : cluster [DBG] pgmap v859: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:29.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:28 smithi193 bash[28559]: cluster 2023-11-17T07:15:27.055352+0000 mgr.smithi097.swvonl (mgr.14180) 1181 : cluster [DBG] pgmap v859: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:30.911 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:30.986 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:13:23.120178Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zohgbz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:30.990 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:30 smithi097 bash[24958]: cluster 2023-11-17T07:15:29.056475+0000 mgr.smithi097.swvonl (mgr.14180) 1182 : cluster [DBG] pgmap v860: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:30 smithi193 bash[28559]: cluster 2023-11-17T07:15:29.056475+0000 mgr.smithi097.swvonl (mgr.14180) 1182 : cluster [DBG] pgmap v860: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:31.792 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:32.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:30.900133+0000 mgr.smithi097.swvonl (mgr.14180) 1183 : audit [DBG] from='client.15020 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:32.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.028753+0000 mon.smithi097 (mon.0) 968 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:15:32.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.338397+0000 mon.smithi097 (mon.0) 969 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:15:32.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.339789+0000 mon.smithi097 (mon.0) 970 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:15:32.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.347879+0000 mon.smithi097 (mon.0) 971 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:32.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.355127+0000 mon.smithi097 (mon.0) 972 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:15:32.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:31 smithi097 bash[24958]: audit 2023-11-17T07:15:31.366649+0000 mon.smithi097 (mon.0) 973 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:32.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:30.900133+0000 mgr.smithi097.swvonl (mgr.14180) 1183 : audit [DBG] from='client.15020 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.028753+0000 mon.smithi097 (mon.0) 968 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.338397+0000 mon.smithi097 (mon.0) 969 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.339789+0000 mon.smithi097 (mon.0) 970 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.347879+0000 mon.smithi097 (mon.0) 971 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.355127+0000 mon.smithi097 (mon.0) 972 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:15:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:31 smithi193 bash[28559]: audit 2023-11-17T07:15:31.366649+0000 mon.smithi097 (mon.0) 973 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:32.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:32.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:32 smithi097 bash[24958]: cluster 2023-11-17T07:15:31.057394+0000 mgr.smithi097.swvonl (mgr.14180) 1184 : cluster [DBG] pgmap v861: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:32.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:32 smithi097 bash[24958]: cluster 2023-11-17T07:15:31.348459+0000 mgr.smithi097.swvonl (mgr.14180) 1185 : cluster [DBG] pgmap v862: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:32.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:32 smithi097 bash[24958]: cephadm 2023-11-17T07:15:31.370080+0000 mgr.smithi097.swvonl (mgr.14180) 1186 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.swmakc on smithi193 2023-11-17T07:15:32.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:32 smithi097 bash[24958]: cluster 2023-11-17T07:15:31.721285+0000 mon.smithi097 (mon.0) 974 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:15:32.994 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:32 smithi097 bash[24958]: cluster 2023-11-17T07:15:31.721364+0000 mon.smithi097 (mon.0) 975 : cluster [INF] Cluster is now healthy 2023-11-17T07:15:33.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:32 smithi193 bash[28559]: cluster 2023-11-17T07:15:31.057394+0000 mgr.smithi097.swvonl (mgr.14180) 1184 : cluster [DBG] pgmap v861: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:32 smithi193 bash[28559]: cluster 2023-11-17T07:15:31.348459+0000 mgr.smithi097.swvonl (mgr.14180) 1185 : cluster [DBG] pgmap v862: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:32 smithi193 bash[28559]: cephadm 2023-11-17T07:15:31.370080+0000 mgr.smithi097.swvonl (mgr.14180) 1186 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.swmakc on smithi193 2023-11-17T07:15:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:32 smithi193 bash[28559]: cluster 2023-11-17T07:15:31.721285+0000 mon.smithi097 (mon.0) 974 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:15:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:32 smithi193 bash[28559]: cluster 2023-11-17T07:15:31.721364+0000 mon.smithi097 (mon.0) 975 : cluster [INF] Cluster is now healthy 2023-11-17T07:15:35.044 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:34 smithi193 bash[28559]: cluster 2023-11-17T07:15:33.349210+0000 mgr.smithi097.swvonl (mgr.14180) 1187 : cluster [DBG] pgmap v863: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:35.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:34 smithi097 bash[24958]: cluster 2023-11-17T07:15:33.349210+0000 mgr.smithi097.swvonl (mgr.14180) 1187 : cluster [DBG] pgmap v863: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:36.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:35 smithi097 bash[24958]: audit 2023-11-17T07:15:35.115192+0000 mon.smithi097 (mon.0) 976 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.swmakc"}]: dispatch 2023-11-17T07:15:36.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:35 smithi193 bash[28559]: audit 2023-11-17T07:15:35.115192+0000 mon.smithi097 (mon.0) 976 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.swmakc"}]: dispatch 2023-11-17T07:15:36.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:36.612 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:27.278355Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jjsbhi on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jjsbhi\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jjsbhi\nDeploy daemon haproxy.nfs.foo.smithi193.jjsbhi ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.018052Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.wwvghf on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-wwvghf\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.wwvghf\nDeploy daemon haproxy.nfs.foo.smithi097.wwvghf ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:14:31.022642Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xlxwzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: cephadm 2023-11-17T07:15:35.114506+0000 mgr.smithi097.swvonl (mgr.14180) 1188 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: cephadm 2023-11-17T07:15:35.114766+0000 mgr.smithi097.swvonl (mgr.14180) 1189 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: cephadm 2023-11-17T07:15:35.116364+0000 mgr.smithi097.swvonl (mgr.14180) 1190 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: cephadm 2023-11-17T07:15:35.120739+0000 mgr.smithi097.swvonl (mgr.14180) 1191 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.txmxks on smithi097 2023-11-17T07:15:37.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:36 smithi097 bash[24958]: cluster 2023-11-17T07:15:35.350037+0000 mgr.smithi097.swvonl (mgr.14180) 1192 : cluster [DBG] pgmap v864: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: cephadm 2023-11-17T07:15:35.114506+0000 mgr.smithi097.swvonl (mgr.14180) 1188 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: cephadm 2023-11-17T07:15:35.114766+0000 mgr.smithi097.swvonl (mgr.14180) 1189 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: cephadm 2023-11-17T07:15:35.116364+0000 mgr.smithi097.swvonl (mgr.14180) 1190 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.swmakc ... 2023-11-17T07:15:37.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: cephadm 2023-11-17T07:15:35.120739+0000 mgr.smithi097.swvonl (mgr.14180) 1191 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.txmxks on smithi097 2023-11-17T07:15:37.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:36 smithi193 bash[28559]: cluster 2023-11-17T07:15:35.350037+0000 mgr.smithi097.swvonl (mgr.14180) 1192 : cluster [DBG] pgmap v864: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:37.505 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:37 smithi097 bash[24958]: audit 2023-11-17T07:15:36.600579+0000 mgr.smithi097.swvonl (mgr.14180) 1193 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:38.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:37 smithi193 bash[28559]: audit 2023-11-17T07:15:36.600579+0000 mgr.smithi097.swvonl (mgr.14180) 1193 : audit [DBG] from='client.15024 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:38.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:38.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:38 smithi097 bash[24958]: cluster 2023-11-17T07:15:37.350786+0000 mgr.smithi097.swvonl (mgr.14180) 1194 : cluster [DBG] pgmap v865: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:39.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:38 smithi193 bash[28559]: cluster 2023-11-17T07:15:37.350786+0000 mgr.smithi097.swvonl (mgr.14180) 1194 : cluster [DBG] pgmap v865: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:39.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:39 smithi097 bash[24958]: audit 2023-11-17T07:15:39.643292+0000 mon.smithi097 (mon.0) 977 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.txmxks"}]: dispatch 2023-11-17T07:15:40.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:39 smithi193 bash[28559]: audit 2023-11-17T07:15:39.643292+0000 mon.smithi097 (mon.0) 977 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.txmxks"}]: dispatch 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cluster 2023-11-17T07:15:39.351559+0000 mgr.smithi097.swvonl (mgr.14180) 1195 : cluster [DBG] pgmap v866: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cephadm 2023-11-17T07:15:39.642281+0000 mgr.smithi097.swvonl (mgr.14180) 1196 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:15:41.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cephadm 2023-11-17T07:15:39.642786+0000 mgr.smithi097.swvonl (mgr.14180) 1197 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cephadm 2023-11-17T07:15:39.644491+0000 mgr.smithi097.swvonl (mgr.14180) 1198 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cephadm 2023-11-17T07:15:39.648444+0000 mgr.smithi097.swvonl (mgr.14180) 1199 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:15:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cephadm 2023-11-17T07:15:39.650686+0000 mgr.smithi097.swvonl (mgr.14180) 1200 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:15:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cluster 2023-11-17T07:15:39.651513+0000 mgr.smithi097.swvonl (mgr.14180) 1201 : cluster [DBG] pgmap v867: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:15:41.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:40 smithi097 bash[24958]: cluster 2023-11-17T07:15:39.764135+0000 mon.smithi097 (mon.0) 978 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:15:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cluster 2023-11-17T07:15:39.351559+0000 mgr.smithi097.swvonl (mgr.14180) 1195 : cluster [DBG] pgmap v866: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:41.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cephadm 2023-11-17T07:15:39.642281+0000 mgr.smithi097.swvonl (mgr.14180) 1196 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:15:41.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cephadm 2023-11-17T07:15:39.642786+0000 mgr.smithi097.swvonl (mgr.14180) 1197 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cephadm 2023-11-17T07:15:39.644491+0000 mgr.smithi097.swvonl (mgr.14180) 1198 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txmxks ... 2023-11-17T07:15:41.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cephadm 2023-11-17T07:15:39.648444+0000 mgr.smithi097.swvonl (mgr.14180) 1199 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cephadm 2023-11-17T07:15:39.650686+0000 mgr.smithi097.swvonl (mgr.14180) 1200 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:15:41.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cluster 2023-11-17T07:15:39.651513+0000 mgr.smithi097.swvonl (mgr.14180) 1201 : cluster [DBG] pgmap v867: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:15:41.064 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:40 smithi193 bash[28559]: cluster 2023-11-17T07:15:39.764135+0000 mon.smithi097 (mon.0) 978 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:15:42.544 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:42.544 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:43.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:42 smithi097 bash[24958]: cluster 2023-11-17T07:15:41.652790+0000 mgr.smithi097.swvonl (mgr.14180) 1202 : cluster [DBG] pgmap v868: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:43.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:42 smithi193 bash[28559]: cluster 2023-11-17T07:15:41.652790+0000 mgr.smithi097.swvonl (mgr.14180) 1202 : cluster [DBG] pgmap v868: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:43.400 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:43.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:43 smithi097 bash[24958]: audit 2023-11-17T07:15:42.533026+0000 mgr.smithi097.swvonl (mgr.14180) 1203 : audit [DBG] from='client.15028 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:44.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:43 smithi193 bash[28559]: audit 2023-11-17T07:15:42.533026+0000 mgr.smithi097.swvonl (mgr.14180) 1203 : audit [DBG] from='client.15028 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:44.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:44.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:44 smithi097 bash[24958]: cluster 2023-11-17T07:15:43.653979+0000 mgr.smithi097.swvonl (mgr.14180) 1204 : cluster [DBG] pgmap v869: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:44.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:44 smithi097 bash[24958]: audit 2023-11-17T07:15:44.178983+0000 mon.smithi097 (mon.0) 979 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:45.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:44 smithi193 bash[28559]: cluster 2023-11-17T07:15:43.653979+0000 mgr.smithi097.swvonl (mgr.14180) 1204 : cluster [DBG] pgmap v869: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:45.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:44 smithi193 bash[28559]: audit 2023-11-17T07:15:44.178983+0000 mon.smithi097 (mon.0) 979 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:15:47.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:46 smithi097 bash[24958]: cluster 2023-11-17T07:15:45.655248+0000 mgr.smithi097.swvonl (mgr.14180) 1205 : cluster [DBG] pgmap v870: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:47.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:46 smithi193 bash[28559]: cluster 2023-11-17T07:15:45.655248+0000 mgr.smithi097.swvonl (mgr.14180) 1205 : cluster [DBG] pgmap v870: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:48.016 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:48.017 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:48.826 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:49.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:48 smithi097 bash[24958]: cluster 2023-11-17T07:15:47.656100+0000 mgr.smithi097.swvonl (mgr.14180) 1206 : cluster [DBG] pgmap v871: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:49.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:48 smithi193 bash[28559]: cluster 2023-11-17T07:15:47.656100+0000 mgr.smithi097.swvonl (mgr.14180) 1206 : cluster [DBG] pgmap v871: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:49.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:50.026 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:49 smithi097 bash[24958]: audit 2023-11-17T07:15:48.004327+0000 mgr.smithi097.swvonl (mgr.14180) 1207 : audit [DBG] from='client.15032 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:50.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:49 smithi193 bash[28559]: audit 2023-11-17T07:15:48.004327+0000 mgr.smithi097.swvonl (mgr.14180) 1207 : audit [DBG] from='client.15032 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:51.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:50 smithi097 bash[24958]: cluster 2023-11-17T07:15:49.657188+0000 mgr.smithi097.swvonl (mgr.14180) 1208 : cluster [DBG] pgmap v872: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:51.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:50 smithi193 bash[28559]: cluster 2023-11-17T07:15:49.657188+0000 mgr.smithi097.swvonl (mgr.14180) 1208 : cluster [DBG] pgmap v872: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:15:52.922 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:52 smithi097 bash[24958]: cluster 2023-11-17T07:15:51.658064+0000 mgr.smithi097.swvonl (mgr.14180) 1209 : cluster [DBG] pgmap v873: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:53.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:52 smithi193 bash[28559]: cluster 2023-11-17T07:15:51.658064+0000 mgr.smithi097.swvonl (mgr.14180) 1209 : cluster [DBG] pgmap v873: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:15:53.356 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:53.356 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:54.275 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:15:55.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:54 smithi097 bash[24958]: audit 2023-11-17T07:15:53.343075+0000 mgr.smithi097.swvonl (mgr.14180) 1210 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:55.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:54 smithi097 bash[24958]: cluster 2023-11-17T07:15:53.659259+0000 mgr.smithi097.swvonl (mgr.14180) 1211 : cluster [DBG] pgmap v874: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:54 smithi193 bash[28559]: audit 2023-11-17T07:15:53.343075+0000 mgr.smithi097.swvonl (mgr.14180) 1210 : audit [DBG] from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:15:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:54 smithi193 bash[28559]: cluster 2023-11-17T07:15:53.659259+0000 mgr.smithi097.swvonl (mgr.14180) 1211 : cluster [DBG] pgmap v874: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:55.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:15:57.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:56 smithi097 bash[24958]: cluster 2023-11-17T07:15:55.660891+0000 mgr.smithi097.swvonl (mgr.14180) 1212 : cluster [DBG] pgmap v875: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:57.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:56 smithi193 bash[28559]: cluster 2023-11-17T07:15:55.660891+0000 mgr.smithi097.swvonl (mgr.14180) 1212 : cluster [DBG] pgmap v875: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:58.973 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:15:58.973 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:15:58.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:15:58 smithi097 bash[24958]: cluster 2023-11-17T07:15:57.661695+0000 mgr.smithi097.swvonl (mgr.14180) 1213 : cluster [DBG] pgmap v876: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:59.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:15:58 smithi193 bash[28559]: cluster 2023-11-17T07:15:57.661695+0000 mgr.smithi097.swvonl (mgr.14180) 1213 : cluster [DBG] pgmap v876: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:15:59.812 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:16:00.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:16:01.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:00 smithi097 bash[24958]: audit 2023-11-17T07:15:58.966534+0000 mgr.smithi097.swvonl (mgr.14180) 1214 : audit [DBG] from='client.15040 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:01.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:00 smithi097 bash[24958]: cluster 2023-11-17T07:15:59.662973+0000 mgr.smithi097.swvonl (mgr.14180) 1215 : cluster [DBG] pgmap v877: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:01.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:00 smithi193 bash[28559]: audit 2023-11-17T07:15:58.966534+0000 mgr.smithi097.swvonl (mgr.14180) 1214 : audit [DBG] from='client.15040 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:01.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:00 smithi193 bash[28559]: cluster 2023-11-17T07:15:59.662973+0000 mgr.smithi097.swvonl (mgr.14180) 1215 : cluster [DBG] pgmap v877: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:02.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:02 smithi097 bash[24958]: cluster 2023-11-17T07:16:01.664188+0000 mgr.smithi097.swvonl (mgr.14180) 1216 : cluster [DBG] pgmap v878: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:03.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:02 smithi193 bash[28559]: cluster 2023-11-17T07:16:01.664188+0000 mgr.smithi097.swvonl (mgr.14180) 1216 : cluster [DBG] pgmap v878: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:04.420 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:16:04.420 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:16:05.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:04 smithi097 bash[24958]: cluster 2023-11-17T07:16:03.665299+0000 mgr.smithi097.swvonl (mgr.14180) 1217 : cluster [DBG] pgmap v879: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:05.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:04 smithi193 bash[28559]: cluster 2023-11-17T07:16:03.665299+0000 mgr.smithi097.swvonl (mgr.14180) 1217 : cluster [DBG] pgmap v879: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:05.339 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:16:06.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:05 smithi097 bash[24958]: audit 2023-11-17T07:16:04.409105+0000 mgr.smithi097.swvonl (mgr.14180) 1218 : audit [DBG] from='client.15044 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:06.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:05 smithi193 bash[28559]: audit 2023-11-17T07:16:04.409105+0000 mgr.smithi097.swvonl (mgr.14180) 1218 : audit [DBG] from='client.15044 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:06.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:16:07.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:06 smithi097 bash[24958]: cluster 2023-11-17T07:16:05.666686+0000 mgr.smithi097.swvonl (mgr.14180) 1219 : cluster [DBG] pgmap v880: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:07.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:06 smithi193 bash[28559]: cluster 2023-11-17T07:16:05.666686+0000 mgr.smithi097.swvonl (mgr.14180) 1219 : cluster [DBG] pgmap v880: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:09.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:08 smithi097 bash[24958]: cluster 2023-11-17T07:16:07.667664+0000 mgr.smithi097.swvonl (mgr.14180) 1220 : cluster [DBG] pgmap v881: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:09.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:08 smithi193 bash[28559]: cluster 2023-11-17T07:16:07.667664+0000 mgr.smithi097.swvonl (mgr.14180) 1220 : cluster [DBG] pgmap v881: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:10.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:16:10.086 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:16:10.928 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:16:11.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:10 smithi097 bash[24958]: cluster 2023-11-17T07:16:09.668835+0000 mgr.smithi097.swvonl (mgr.14180) 1221 : cluster [DBG] pgmap v882: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:11.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:10 smithi193 bash[28559]: cluster 2023-11-17T07:16:09.668835+0000 mgr.smithi097.swvonl (mgr.14180) 1221 : cluster [DBG] pgmap v882: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:11.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:16:12.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:11 smithi097 bash[24958]: audit 2023-11-17T07:16:10.074423+0000 mgr.smithi097.swvonl (mgr.14180) 1222 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:11 smithi193 bash[28559]: audit 2023-11-17T07:16:10.074423+0000 mgr.smithi097.swvonl (mgr.14180) 1222 : audit [DBG] from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:13.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:12 smithi097 bash[24958]: cluster 2023-11-17T07:16:11.670169+0000 mgr.smithi097.swvonl (mgr.14180) 1223 : cluster [DBG] pgmap v883: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:13.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:12 smithi193 bash[28559]: cluster 2023-11-17T07:16:11.670169+0000 mgr.smithi097.swvonl (mgr.14180) 1223 : cluster [DBG] pgmap v883: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:15.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:14 smithi097 bash[24958]: cluster 2023-11-17T07:16:13.671229+0000 mgr.smithi097.swvonl (mgr.14180) 1224 : cluster [DBG] pgmap v884: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:14 smithi193 bash[28559]: cluster 2023-11-17T07:16:13.671229+0000 mgr.smithi097.swvonl (mgr.14180) 1224 : cluster [DBG] pgmap v884: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:15.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:16:15.514 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:16:16.361 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:16:17.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:16 smithi097 bash[24958]: audit 2023-11-17T07:16:15.502944+0000 mgr.smithi097.swvonl (mgr.14180) 1225 : audit [DBG] from='client.15052 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:17.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:16 smithi097 bash[24958]: cluster 2023-11-17T07:16:15.672580+0000 mgr.smithi097.swvonl (mgr.14180) 1226 : cluster [DBG] pgmap v885: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:17.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:16 smithi193 bash[28559]: audit 2023-11-17T07:16:15.502944+0000 mgr.smithi097.swvonl (mgr.14180) 1225 : audit [DBG] from='client.15052 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:17.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:16 smithi193 bash[28559]: cluster 2023-11-17T07:16:15.672580+0000 mgr.smithi097.swvonl (mgr.14180) 1226 : cluster [DBG] pgmap v885: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:17.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:16:19.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:18 smithi097 bash[24958]: cluster 2023-11-17T07:16:17.673620+0000 mgr.smithi097.swvonl (mgr.14180) 1227 : cluster [DBG] pgmap v886: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:19.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:18 smithi193 bash[28559]: cluster 2023-11-17T07:16:17.673620+0000 mgr.smithi097.swvonl (mgr.14180) 1227 : cluster [DBG] pgmap v886: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:20.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:16:20.941 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:16:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:20 smithi097 bash[24958]: cluster 2023-11-17T07:16:19.674704+0000 mgr.smithi097.swvonl (mgr.14180) 1228 : cluster [DBG] pgmap v887: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:21.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:20 smithi193 bash[28559]: cluster 2023-11-17T07:16:19.674704+0000 mgr.smithi097.swvonl (mgr.14180) 1228 : cluster [DBG] pgmap v887: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:21.759 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:16:22.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:21 smithi097 bash[24958]: audit 2023-11-17T07:16:20.929661+0000 mgr.smithi097.swvonl (mgr.14180) 1229 : audit [DBG] from='client.15056 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:22.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:21 smithi193 bash[28559]: audit 2023-11-17T07:16:20.929661+0000 mgr.smithi097.swvonl (mgr.14180) 1229 : audit [DBG] from='client.15056 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:16:22.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:16:23.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:22 smithi097 bash[24958]: cluster 2023-11-17T07:16:21.675904+0000 mgr.smithi097.swvonl (mgr.14180) 1230 : cluster [DBG] pgmap v888: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:23.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:22 smithi193 bash[28559]: cluster 2023-11-17T07:16:21.675904+0000 mgr.smithi097.swvonl (mgr.14180) 1230 : cluster [DBG] pgmap v888: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:16:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:24 smithi097 bash[24958]: cluster 2023-11-17T07:16:23.677134+0000 mgr.smithi097.swvonl (mgr.14180) 1231 : cluster [DBG] pgmap v889: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:25.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:24 smithi193 bash[28559]: cluster 2023-11-17T07:16:23.677134+0000 mgr.smithi097.swvonl (mgr.14180) 1231 : cluster [DBG] pgmap v889: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:16:26.448 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:18:31.385 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:14:31.024484Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kolujy on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:35.116138Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.swmakc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-swmakc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.swmakc\nDeploy daemon haproxy.nfs.foo.smithi193.swmakc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.644271Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txmxks on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txmxks\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txmxks\nDeploy daemon haproxy.nfs.foo.smithi097.txmxks ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:15:39.648248Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hhwzpi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:15:39.650553Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ertbzz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:18:31.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:26 smithi097 bash[24958]: cluster 2023-11-17T07:16:25.677957+0000 mgr.smithi097.swvonl (mgr.14180) 1232 : cluster [DBG] pgmap v890: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.399 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:26 smithi193 bash[28559]: cluster 2023-11-17T07:16:25.677957+0000 mgr.smithi097.swvonl (mgr.14180) 1232 : cluster [DBG] pgmap v890: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:27 smithi193 bash[28559]: audit 2023-11-17T07:16:26.436814+0000 mgr.smithi097.swvonl (mgr.14180) 1233 : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:28 smithi193 bash[28559]: cluster 2023-11-17T07:16:27.678919+0000 mgr.smithi097.swvonl (mgr.14180) 1234 : cluster [DBG] pgmap v891: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:30 smithi193 bash[28559]: cluster 2023-11-17T07:16:29.679965+0000 mgr.smithi097.swvonl (mgr.14180) 1235 : cluster [DBG] pgmap v892: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:32 smithi193 bash[28559]: cluster 2023-11-17T07:16:31.681356+0000 mgr.smithi097.swvonl (mgr.14180) 1236 : cluster [DBG] pgmap v893: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.411 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:34 smithi193 bash[28559]: cluster 2023-11-17T07:16:33.682423+0000 mgr.smithi097.swvonl (mgr.14180) 1237 : cluster [DBG] pgmap v894: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:36 smithi193 bash[28559]: cluster 2023-11-17T07:16:35.683629+0000 mgr.smithi097.swvonl (mgr.14180) 1238 : cluster [DBG] pgmap v895: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:38 smithi193 bash[28559]: cluster 2023-11-17T07:16:37.684608+0000 mgr.smithi097.swvonl (mgr.14180) 1239 : cluster [DBG] pgmap v896: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:39 smithi193 bash[28559]: audit 2023-11-17T07:16:39.655314+0000 mon.smithi097 (mon.0) 980 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: cluster 2023-11-17T07:16:39.685433+0000 mgr.smithi097.swvonl (mgr.14180) 1240 : cluster [DBG] pgmap v897: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: audit 2023-11-17T07:16:39.964286+0000 mon.smithi097 (mon.0) 981 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: audit 2023-11-17T07:16:39.965741+0000 mon.smithi097 (mon.0) 982 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: audit 2023-11-17T07:16:39.974005+0000 mon.smithi097 (mon.0) 983 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: audit 2023-11-17T07:16:39.982000+0000 mon.smithi097 (mon.0) 984 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:40 smithi193 bash[28559]: audit 2023-11-17T07:16:39.997446+0000 mon.smithi097 (mon.0) 985 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:41 smithi193 bash[28559]: cluster 2023-11-17T07:16:39.975110+0000 mgr.smithi097.swvonl (mgr.14180) 1241 : cluster [DBG] pgmap v898: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 397 B/s rd, 0 op/s 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:41 smithi193 bash[28559]: cephadm 2023-11-17T07:16:40.000976+0000 mgr.smithi097.swvonl (mgr.14180) 1242 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kaswny on smithi193 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:41 smithi193 bash[28559]: cluster 2023-11-17T07:16:40.970857+0000 mon.smithi097 (mon.0) 986 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:41 smithi193 bash[28559]: cluster 2023-11-17T07:16:40.970906+0000 mon.smithi097 (mon.0) 987 : cluster [INF] Cluster is now healthy 2023-11-17T07:18:31.412 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:41 smithi193 bash[28559]: audit 2023-11-17T07:16:41.544054+0000 mon.smithi097 (mon.0) 988 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:43 smithi193 bash[28559]: cluster 2023-11-17T07:16:41.975867+0000 mgr.smithi097.swvonl (mgr.14180) 1243 : cluster [DBG] pgmap v899: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: cephadm 2023-11-17T07:16:43.916209+0000 mgr.smithi097.swvonl (mgr.14180) 1244 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.413 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.414 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: cephadm 2023-11-17T07:16:43.916695+0000 mgr.smithi097.swvonl (mgr.14180) 1245 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: audit 2023-11-17T07:16:43.917401+0000 mon.smithi097 (mon.0) 989 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kaswny"}]: dispatch 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: cephadm 2023-11-17T07:16:43.919132+0000 mgr.smithi097.swvonl (mgr.14180) 1246 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kaswny on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.415 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:44 smithi193 bash[28559]: cephadm 2023-11-17T07:16:43.922651+0000 mgr.smithi097.swvonl (mgr.14180) 1247 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.dkvfaq on smithi097 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:45 smithi193 bash[28559]: cluster 2023-11-17T07:16:43.976701+0000 mgr.smithi097.swvonl (mgr.14180) 1248 : cluster [DBG] pgmap v900: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:47 smithi193 bash[28559]: cluster 2023-11-17T07:16:45.977784+0000 mgr.smithi097.swvonl (mgr.14180) 1249 : cluster [DBG] pgmap v901: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:48 smithi193 bash[28559]: audit 2023-11-17T07:16:48.241944+0000 mon.smithi097 (mon.0) 990 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.dkvfaq"}]: dispatch 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cluster 2023-11-17T07:16:47.978358+0000 mgr.smithi097.swvonl (mgr.14180) 1250 : cluster [DBG] pgmap v902: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cephadm 2023-11-17T07:16:48.240789+0000 mgr.smithi097.swvonl (mgr.14180) 1251 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.416 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.417 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cephadm 2023-11-17T07:16:48.241266+0000 mgr.smithi097.swvonl (mgr.14180) 1252 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cephadm 2023-11-17T07:16:48.243296+0000 mgr.smithi097.swvonl (mgr.14180) 1253 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.dkvfaq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.419 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.421 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.421 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cephadm 2023-11-17T07:16:48.246387+0000 mgr.smithi097.swvonl (mgr.14180) 1254 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ejlspq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.421 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cephadm 2023-11-17T07:16:48.248958+0000 mgr.smithi097.swvonl (mgr.14180) 1255 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.421 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cluster 2023-11-17T07:16:48.250413+0000 mgr.smithi097.swvonl (mgr.14180) 1256 : cluster [DBG] pgmap v903: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: cluster 2023-11-17T07:16:48.790635+0000 mon.smithi097 (mon.0) 991 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:49 smithi193 bash[28559]: audit 2023-11-17T07:16:49.226904+0000 mon.smithi097 (mon.0) 992 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:51 smithi193 bash[28559]: cluster 2023-11-17T07:16:50.251673+0000 mgr.smithi097.swvonl (mgr.14180) 1257 : cluster [DBG] pgmap v904: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:53 smithi193 bash[28559]: cluster 2023-11-17T07:16:52.252624+0000 mgr.smithi097.swvonl (mgr.14180) 1258 : cluster [DBG] pgmap v905: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:55 smithi193 bash[28559]: cluster 2023-11-17T07:16:54.253761+0000 mgr.smithi097.swvonl (mgr.14180) 1259 : cluster [DBG] pgmap v906: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:57 smithi193 bash[28559]: cluster 2023-11-17T07:16:56.255186+0000 mgr.smithi097.swvonl (mgr.14180) 1260 : cluster [DBG] pgmap v907: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:57 smithi193 bash[28559]: audit 2023-11-17T07:16:56.544770+0000 mon.smithi097 (mon.0) 993 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:16:59 smithi193 bash[28559]: cluster 2023-11-17T07:16:58.256361+0000 mgr.smithi097.swvonl (mgr.14180) 1261 : cluster [DBG] pgmap v908: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:01 smithi193 bash[28559]: cluster 2023-11-17T07:17:00.257860+0000 mgr.smithi097.swvonl (mgr.14180) 1262 : cluster [DBG] pgmap v909: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:03 smithi193 bash[28559]: cluster 2023-11-17T07:17:02.258882+0000 mgr.smithi097.swvonl (mgr.14180) 1263 : cluster [DBG] pgmap v910: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:05 smithi193 bash[28559]: cluster 2023-11-17T07:17:04.260052+0000 mgr.smithi097.swvonl (mgr.14180) 1264 : cluster [DBG] pgmap v911: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:07 smithi193 bash[28559]: cluster 2023-11-17T07:17:06.261410+0000 mgr.smithi097.swvonl (mgr.14180) 1265 : cluster [DBG] pgmap v912: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.422 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:09 smithi193 bash[28559]: cluster 2023-11-17T07:17:08.262458+0000 mgr.smithi097.swvonl (mgr.14180) 1266 : cluster [DBG] pgmap v913: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:11 smithi193 bash[28559]: cluster 2023-11-17T07:17:10.263812+0000 mgr.smithi097.swvonl (mgr.14180) 1267 : cluster [DBG] pgmap v914: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:13 smithi193 bash[28559]: cluster 2023-11-17T07:17:12.264805+0000 mgr.smithi097.swvonl (mgr.14180) 1268 : cluster [DBG] pgmap v915: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:15 smithi193 bash[28559]: cluster 2023-11-17T07:17:14.265912+0000 mgr.smithi097.swvonl (mgr.14180) 1269 : cluster [DBG] pgmap v916: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:17 smithi193 bash[28559]: cluster 2023-11-17T07:17:16.267243+0000 mgr.smithi097.swvonl (mgr.14180) 1270 : cluster [DBG] pgmap v917: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:19 smithi193 bash[28559]: cluster 2023-11-17T07:17:18.268304+0000 mgr.smithi097.swvonl (mgr.14180) 1271 : cluster [DBG] pgmap v918: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:21 smithi193 bash[28559]: cluster 2023-11-17T07:17:20.269729+0000 mgr.smithi097.swvonl (mgr.14180) 1272 : cluster [DBG] pgmap v919: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:23 smithi193 bash[28559]: cluster 2023-11-17T07:17:22.270739+0000 mgr.smithi097.swvonl (mgr.14180) 1273 : cluster [DBG] pgmap v920: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:25 smithi193 bash[28559]: cluster 2023-11-17T07:17:24.271761+0000 mgr.smithi097.swvonl (mgr.14180) 1274 : cluster [DBG] pgmap v921: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:27 smithi193 bash[28559]: cluster 2023-11-17T07:17:26.273242+0000 mgr.smithi097.swvonl (mgr.14180) 1275 : cluster [DBG] pgmap v922: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:29 smithi193 bash[28559]: cluster 2023-11-17T07:17:28.274335+0000 mgr.smithi097.swvonl (mgr.14180) 1276 : cluster [DBG] pgmap v923: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:31 smithi193 bash[28559]: cluster 2023-11-17T07:17:30.275865+0000 mgr.smithi097.swvonl (mgr.14180) 1277 : cluster [DBG] pgmap v924: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:33 smithi193 bash[28559]: cluster 2023-11-17T07:17:32.276908+0000 mgr.smithi097.swvonl (mgr.14180) 1278 : cluster [DBG] pgmap v925: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:35 smithi193 bash[28559]: cluster 2023-11-17T07:17:34.277963+0000 mgr.smithi097.swvonl (mgr.14180) 1279 : cluster [DBG] pgmap v926: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:37 smithi193 bash[28559]: cluster 2023-11-17T07:17:36.279291+0000 mgr.smithi097.swvonl (mgr.14180) 1280 : cluster [DBG] pgmap v927: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:39 smithi193 bash[28559]: cluster 2023-11-17T07:17:38.280281+0000 mgr.smithi097.swvonl (mgr.14180) 1281 : cluster [DBG] pgmap v928: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:41 smithi193 bash[28559]: cluster 2023-11-17T07:17:40.281717+0000 mgr.smithi097.swvonl (mgr.14180) 1282 : cluster [DBG] pgmap v929: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:43 smithi193 bash[28559]: cluster 2023-11-17T07:17:42.282799+0000 mgr.smithi097.swvonl (mgr.14180) 1283 : cluster [DBG] pgmap v930: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:45 smithi193 bash[28559]: cluster 2023-11-17T07:17:44.284034+0000 mgr.smithi097.swvonl (mgr.14180) 1284 : cluster [DBG] pgmap v931: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:47 smithi193 bash[28559]: cluster 2023-11-17T07:17:46.285389+0000 mgr.smithi097.swvonl (mgr.14180) 1285 : cluster [DBG] pgmap v932: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:48 smithi193 bash[28559]: audit 2023-11-17T07:17:48.254840+0000 mon.smithi097 (mon.0) 994 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:48 smithi193 bash[28559]: audit 2023-11-17T07:17:48.528239+0000 mon.smithi097 (mon.0) 995 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: cluster 2023-11-17T07:17:48.286420+0000 mgr.smithi097.swvonl (mgr.14180) 1286 : cluster [DBG] pgmap v933: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: audit 2023-11-17T07:17:48.798134+0000 mon.smithi097 (mon.0) 996 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: audit 2023-11-17T07:17:48.799393+0000 mon.smithi097 (mon.0) 997 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: audit 2023-11-17T07:17:48.806884+0000 mon.smithi097 (mon.0) 998 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: cluster 2023-11-17T07:17:48.807843+0000 mgr.smithi097.swvonl (mgr.14180) 1287 : cluster [DBG] pgmap v934: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.424 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: audit 2023-11-17T07:17:48.815729+0000 mon.smithi097 (mon.0) 999 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: audit 2023-11-17T07:17:48.830752+0000 mon.smithi097 (mon.0) 1000 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:49 smithi193 bash[28559]: cephadm 2023-11-17T07:17:48.834378+0000 mgr.smithi097.swvonl (mgr.14180) 1288 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.yutmgg on smithi193 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:50 smithi193 bash[28559]: cluster 2023-11-17T07:17:49.804072+0000 mon.smithi097 (mon.0) 1001 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:50 smithi193 bash[28559]: cluster 2023-11-17T07:17:49.804145+0000 mon.smithi097 (mon.0) 1002 : cluster [INF] Cluster is now healthy 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:51 smithi193 bash[28559]: cluster 2023-11-17T07:17:50.808916+0000 mgr.smithi097.swvonl (mgr.14180) 1289 : cluster [DBG] pgmap v935: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:52 smithi193 bash[28559]: audit 2023-11-17T07:17:52.641988+0000 mon.smithi097 (mon.0) 1003 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.yutmgg"}]: dispatch 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: cephadm 2023-11-17T07:17:52.640776+0000 mgr.smithi097.swvonl (mgr.14180) 1290 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.425 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.426 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: cephadm 2023-11-17T07:17:52.641292+0000 mgr.smithi097.swvonl (mgr.14180) 1291 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: cephadm 2023-11-17T07:17:52.643738+0000 mgr.smithi097.swvonl (mgr.14180) 1292 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.427 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: cephadm 2023-11-17T07:17:52.649265+0000 mgr.smithi097.swvonl (mgr.14180) 1293 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ztkpag on smithi097 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:53 smithi193 bash[28559]: cluster 2023-11-17T07:17:52.809446+0000 mgr.smithi097.swvonl (mgr.14180) 1294 : cluster [DBG] pgmap v936: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.428 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:55 smithi193 bash[28559]: cluster 2023-11-17T07:17:54.810457+0000 mgr.smithi097.swvonl (mgr.14180) 1295 : cluster [DBG] pgmap v937: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:56 smithi193 bash[28559]: audit 2023-11-17T07:17:56.442616+0000 mon.smithi097 (mon.0) 1004 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ztkpag"}]: dispatch 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:56 smithi193 bash[28559]: audit 2023-11-17T07:17:56.546128+0000 mon.smithi097 (mon.0) 1005 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cephadm 2023-11-17T07:17:56.441774+0000 mgr.smithi097.swvonl (mgr.14180) 1296 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.429 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.430 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cephadm 2023-11-17T07:17:56.441997+0000 mgr.smithi097.swvonl (mgr.14180) 1297 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cephadm 2023-11-17T07:17:56.443819+0000 mgr.smithi097.swvonl (mgr.14180) 1298 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.431 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cephadm 2023-11-17T07:17:56.445856+0000 mgr.smithi097.swvonl (mgr.14180) 1299 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cephadm 2023-11-17T07:17:56.447710+0000 mgr.smithi097.swvonl (mgr.14180) 1300 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cluster 2023-11-17T07:17:56.448941+0000 mgr.smithi097.swvonl (mgr.14180) 1301 : cluster [DBG] pgmap v938: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 201 B/s rd, 0 op/s 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:57 smithi193 bash[28559]: cluster 2023-11-17T07:17:56.867875+0000 mon.smithi097 (mon.0) 1006 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:59 smithi193 bash[28559]: cluster 2023-11-17T07:17:58.449866+0000 mgr.smithi097.swvonl (mgr.14180) 1302 : cluster [DBG] pgmap v939: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 201 B/s rd, 0 op/s 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:17:59 smithi193 bash[28559]: audit 2023-11-17T07:17:59.280697+0000 mon.smithi097 (mon.0) 1007 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:01 smithi193 bash[28559]: cluster 2023-11-17T07:18:00.451341+0000 mgr.smithi097.swvonl (mgr.14180) 1303 : cluster [DBG] pgmap v940: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2023-11-17T07:18:31.432 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:03 smithi193 bash[28559]: cluster 2023-11-17T07:18:02.452388+0000 mgr.smithi097.swvonl (mgr.14180) 1304 : cluster [DBG] pgmap v941: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:05 smithi193 bash[28559]: cluster 2023-11-17T07:18:04.453504+0000 mgr.smithi097.swvonl (mgr.14180) 1305 : cluster [DBG] pgmap v942: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:07 smithi193 bash[28559]: cluster 2023-11-17T07:18:06.454822+0000 mgr.smithi097.swvonl (mgr.14180) 1306 : cluster [DBG] pgmap v943: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:09 smithi193 bash[28559]: cluster 2023-11-17T07:18:08.455894+0000 mgr.smithi097.swvonl (mgr.14180) 1307 : cluster [DBG] pgmap v944: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:11 smithi193 bash[28559]: cluster 2023-11-17T07:18:10.457650+0000 mgr.smithi097.swvonl (mgr.14180) 1308 : cluster [DBG] pgmap v945: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:11 smithi193 bash[28559]: audit 2023-11-17T07:18:11.546620+0000 mon.smithi097 (mon.0) 1008 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:13 smithi193 bash[28559]: cluster 2023-11-17T07:18:12.458684+0000 mgr.smithi097.swvonl (mgr.14180) 1309 : cluster [DBG] pgmap v946: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:15 smithi193 bash[28559]: cluster 2023-11-17T07:18:14.459828+0000 mgr.smithi097.swvonl (mgr.14180) 1310 : cluster [DBG] pgmap v947: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:17 smithi193 bash[28559]: cluster 2023-11-17T07:18:16.461340+0000 mgr.smithi097.swvonl (mgr.14180) 1311 : cluster [DBG] pgmap v948: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:19 smithi193 bash[28559]: cluster 2023-11-17T07:18:18.462533+0000 mgr.smithi097.swvonl (mgr.14180) 1312 : cluster [DBG] pgmap v949: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:21 smithi193 bash[28559]: cluster 2023-11-17T07:18:20.464097+0000 mgr.smithi097.swvonl (mgr.14180) 1313 : cluster [DBG] pgmap v950: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:23 smithi193 bash[28559]: cluster 2023-11-17T07:18:22.465213+0000 mgr.smithi097.swvonl (mgr.14180) 1314 : cluster [DBG] pgmap v951: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:25 smithi193 bash[28559]: cluster 2023-11-17T07:18:24.466381+0000 mgr.smithi097.swvonl (mgr.14180) 1315 : cluster [DBG] pgmap v952: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:27 smithi193 bash[28559]: cluster 2023-11-17T07:18:26.467682+0000 mgr.smithi097.swvonl (mgr.14180) 1316 : cluster [DBG] pgmap v953: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.433 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:29 smithi193 bash[28559]: cluster 2023-11-17T07:18:28.468694+0000 mgr.smithi097.swvonl (mgr.14180) 1317 : cluster [DBG] pgmap v954: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.434 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:27 smithi097 bash[24958]: audit 2023-11-17T07:16:26.436814+0000 mgr.smithi097.swvonl (mgr.14180) 1233 : audit [DBG] from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:31.434 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:28 smithi097 bash[24958]: cluster 2023-11-17T07:16:27.678919+0000 mgr.smithi097.swvonl (mgr.14180) 1234 : cluster [DBG] pgmap v891: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.434 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:30 smithi097 bash[24958]: cluster 2023-11-17T07:16:29.679965+0000 mgr.smithi097.swvonl (mgr.14180) 1235 : cluster [DBG] pgmap v892: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.434 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:32 smithi097 bash[24958]: cluster 2023-11-17T07:16:31.681356+0000 mgr.smithi097.swvonl (mgr.14180) 1236 : cluster [DBG] pgmap v893: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.434 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:34 smithi097 bash[24958]: cluster 2023-11-17T07:16:33.682423+0000 mgr.smithi097.swvonl (mgr.14180) 1237 : cluster [DBG] pgmap v894: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:36 smithi097 bash[24958]: cluster 2023-11-17T07:16:35.683629+0000 mgr.smithi097.swvonl (mgr.14180) 1238 : cluster [DBG] pgmap v895: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:38 smithi097 bash[24958]: cluster 2023-11-17T07:16:37.684608+0000 mgr.smithi097.swvonl (mgr.14180) 1239 : cluster [DBG] pgmap v896: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:39 smithi097 bash[24958]: audit 2023-11-17T07:16:39.655314+0000 mon.smithi097 (mon.0) 980 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: cluster 2023-11-17T07:16:39.685433+0000 mgr.smithi097.swvonl (mgr.14180) 1240 : cluster [DBG] pgmap v897: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: audit 2023-11-17T07:16:39.964286+0000 mon.smithi097 (mon.0) 981 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: audit 2023-11-17T07:16:39.965741+0000 mon.smithi097 (mon.0) 982 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: audit 2023-11-17T07:16:39.974005+0000 mon.smithi097 (mon.0) 983 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: audit 2023-11-17T07:16:39.982000+0000 mon.smithi097 (mon.0) 984 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:40 smithi097 bash[24958]: audit 2023-11-17T07:16:39.997446+0000 mon.smithi097 (mon.0) 985 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:41 smithi097 bash[24958]: cluster 2023-11-17T07:16:39.975110+0000 mgr.smithi097.swvonl (mgr.14180) 1241 : cluster [DBG] pgmap v898: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 397 B/s rd, 0 op/s 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:41 smithi097 bash[24958]: cephadm 2023-11-17T07:16:40.000976+0000 mgr.smithi097.swvonl (mgr.14180) 1242 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kaswny on smithi193 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:41 smithi097 bash[24958]: cluster 2023-11-17T07:16:40.970857+0000 mon.smithi097 (mon.0) 986 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:41 smithi097 bash[24958]: cluster 2023-11-17T07:16:40.970906+0000 mon.smithi097 (mon.0) 987 : cluster [INF] Cluster is now healthy 2023-11-17T07:18:31.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:41 smithi097 bash[24958]: audit 2023-11-17T07:16:41.544054+0000 mon.smithi097 (mon.0) 988 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:43 smithi097 bash[24958]: cluster 2023-11-17T07:16:41.975867+0000 mgr.smithi097.swvonl (mgr.14180) 1243 : cluster [DBG] pgmap v899: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: cephadm 2023-11-17T07:16:43.916209+0000 mgr.smithi097.swvonl (mgr.14180) 1244 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.436 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.437 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: cephadm 2023-11-17T07:16:43.916695+0000 mgr.smithi097.swvonl (mgr.14180) 1245 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: audit 2023-11-17T07:16:43.917401+0000 mon.smithi097 (mon.0) 989 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kaswny"}]: dispatch 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: cephadm 2023-11-17T07:16:43.919132+0000 mgr.smithi097.swvonl (mgr.14180) 1246 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kaswny on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kaswny 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kaswny 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kaswny ... 2023-11-17T07:18:31.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:44 smithi097 bash[24958]: cephadm 2023-11-17T07:16:43.922651+0000 mgr.smithi097.swvonl (mgr.14180) 1247 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.dkvfaq on smithi097 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:45 smithi097 bash[24958]: cluster 2023-11-17T07:16:43.976701+0000 mgr.smithi097.swvonl (mgr.14180) 1248 : cluster [DBG] pgmap v900: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:47 smithi097 bash[24958]: cluster 2023-11-17T07:16:45.977784+0000 mgr.smithi097.swvonl (mgr.14180) 1249 : cluster [DBG] pgmap v901: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:48 smithi097 bash[24958]: audit 2023-11-17T07:16:48.241944+0000 mon.smithi097 (mon.0) 990 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.dkvfaq"}]: dispatch 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cluster 2023-11-17T07:16:47.978358+0000 mgr.smithi097.swvonl (mgr.14180) 1250 : cluster [DBG] pgmap v902: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cephadm 2023-11-17T07:16:48.240789+0000 mgr.smithi097.swvonl (mgr.14180) 1251 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:18:31.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cephadm 2023-11-17T07:16:48.241266+0000 mgr.smithi097.swvonl (mgr.14180) 1252 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cephadm 2023-11-17T07:16:48.243296+0000 mgr.smithi097.swvonl (mgr.14180) 1253 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.dkvfaq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-dkvfaq 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.dkvfaq 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.dkvfaq ... 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:18:31.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cephadm 2023-11-17T07:16:48.246387+0000 mgr.smithi097.swvonl (mgr.14180) 1254 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ejlspq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cephadm 2023-11-17T07:16:48.248958+0000 mgr.smithi097.swvonl (mgr.14180) 1255 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cluster 2023-11-17T07:16:48.250413+0000 mgr.smithi097.swvonl (mgr.14180) 1256 : cluster [DBG] pgmap v903: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 B/s rd, 0 op/s 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: cluster 2023-11-17T07:16:48.790635+0000 mon.smithi097 (mon.0) 991 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:49 smithi097 bash[24958]: audit 2023-11-17T07:16:49.226904+0000 mon.smithi097 (mon.0) 992 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:51 smithi097 bash[24958]: cluster 2023-11-17T07:16:50.251673+0000 mgr.smithi097.swvonl (mgr.14180) 1257 : cluster [DBG] pgmap v904: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:53 smithi097 bash[24958]: cluster 2023-11-17T07:16:52.252624+0000 mgr.smithi097.swvonl (mgr.14180) 1258 : cluster [DBG] pgmap v905: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:55 smithi097 bash[24958]: cluster 2023-11-17T07:16:54.253761+0000 mgr.smithi097.swvonl (mgr.14180) 1259 : cluster [DBG] pgmap v906: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:57 smithi097 bash[24958]: cluster 2023-11-17T07:16:56.255186+0000 mgr.smithi097.swvonl (mgr.14180) 1260 : cluster [DBG] pgmap v907: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:57 smithi097 bash[24958]: audit 2023-11-17T07:16:56.544770+0000 mon.smithi097 (mon.0) 993 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:16:59 smithi097 bash[24958]: cluster 2023-11-17T07:16:58.256361+0000 mgr.smithi097.swvonl (mgr.14180) 1261 : cluster [DBG] pgmap v908: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:01 smithi097 bash[24958]: cluster 2023-11-17T07:17:00.257860+0000 mgr.smithi097.swvonl (mgr.14180) 1262 : cluster [DBG] pgmap v909: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:03 smithi097 bash[24958]: cluster 2023-11-17T07:17:02.258882+0000 mgr.smithi097.swvonl (mgr.14180) 1263 : cluster [DBG] pgmap v910: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:05 smithi097 bash[24958]: cluster 2023-11-17T07:17:04.260052+0000 mgr.smithi097.swvonl (mgr.14180) 1264 : cluster [DBG] pgmap v911: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:07 smithi097 bash[24958]: cluster 2023-11-17T07:17:06.261410+0000 mgr.smithi097.swvonl (mgr.14180) 1265 : cluster [DBG] pgmap v912: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:09 smithi097 bash[24958]: cluster 2023-11-17T07:17:08.262458+0000 mgr.smithi097.swvonl (mgr.14180) 1266 : cluster [DBG] pgmap v913: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:11 smithi097 bash[24958]: cluster 2023-11-17T07:17:10.263812+0000 mgr.smithi097.swvonl (mgr.14180) 1267 : cluster [DBG] pgmap v914: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:13 smithi097 bash[24958]: cluster 2023-11-17T07:17:12.264805+0000 mgr.smithi097.swvonl (mgr.14180) 1268 : cluster [DBG] pgmap v915: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:15 smithi097 bash[24958]: cluster 2023-11-17T07:17:14.265912+0000 mgr.smithi097.swvonl (mgr.14180) 1269 : cluster [DBG] pgmap v916: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:17 smithi097 bash[24958]: cluster 2023-11-17T07:17:16.267243+0000 mgr.smithi097.swvonl (mgr.14180) 1270 : cluster [DBG] pgmap v917: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:19 smithi097 bash[24958]: cluster 2023-11-17T07:17:18.268304+0000 mgr.smithi097.swvonl (mgr.14180) 1271 : cluster [DBG] pgmap v918: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:21 smithi097 bash[24958]: cluster 2023-11-17T07:17:20.269729+0000 mgr.smithi097.swvonl (mgr.14180) 1272 : cluster [DBG] pgmap v919: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:23 smithi097 bash[24958]: cluster 2023-11-17T07:17:22.270739+0000 mgr.smithi097.swvonl (mgr.14180) 1273 : cluster [DBG] pgmap v920: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:25 smithi097 bash[24958]: cluster 2023-11-17T07:17:24.271761+0000 mgr.smithi097.swvonl (mgr.14180) 1274 : cluster [DBG] pgmap v921: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:27 smithi097 bash[24958]: cluster 2023-11-17T07:17:26.273242+0000 mgr.smithi097.swvonl (mgr.14180) 1275 : cluster [DBG] pgmap v922: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:29 smithi097 bash[24958]: cluster 2023-11-17T07:17:28.274335+0000 mgr.smithi097.swvonl (mgr.14180) 1276 : cluster [DBG] pgmap v923: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:31 smithi097 bash[24958]: cluster 2023-11-17T07:17:30.275865+0000 mgr.smithi097.swvonl (mgr.14180) 1277 : cluster [DBG] pgmap v924: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:33 smithi097 bash[24958]: cluster 2023-11-17T07:17:32.276908+0000 mgr.smithi097.swvonl (mgr.14180) 1278 : cluster [DBG] pgmap v925: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:35 smithi097 bash[24958]: cluster 2023-11-17T07:17:34.277963+0000 mgr.smithi097.swvonl (mgr.14180) 1279 : cluster [DBG] pgmap v926: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:37 smithi097 bash[24958]: cluster 2023-11-17T07:17:36.279291+0000 mgr.smithi097.swvonl (mgr.14180) 1280 : cluster [DBG] pgmap v927: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:39 smithi097 bash[24958]: cluster 2023-11-17T07:17:38.280281+0000 mgr.smithi097.swvonl (mgr.14180) 1281 : cluster [DBG] pgmap v928: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:41 smithi097 bash[24958]: cluster 2023-11-17T07:17:40.281717+0000 mgr.smithi097.swvonl (mgr.14180) 1282 : cluster [DBG] pgmap v929: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:43 smithi097 bash[24958]: cluster 2023-11-17T07:17:42.282799+0000 mgr.smithi097.swvonl (mgr.14180) 1283 : cluster [DBG] pgmap v930: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:45 smithi097 bash[24958]: cluster 2023-11-17T07:17:44.284034+0000 mgr.smithi097.swvonl (mgr.14180) 1284 : cluster [DBG] pgmap v931: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:47 smithi097 bash[24958]: cluster 2023-11-17T07:17:46.285389+0000 mgr.smithi097.swvonl (mgr.14180) 1285 : cluster [DBG] pgmap v932: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:48 smithi097 bash[24958]: audit 2023-11-17T07:17:48.254840+0000 mon.smithi097 (mon.0) 994 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:48 smithi097 bash[24958]: audit 2023-11-17T07:17:48.528239+0000 mon.smithi097 (mon.0) 995 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: cluster 2023-11-17T07:17:48.286420+0000 mgr.smithi097.swvonl (mgr.14180) 1286 : cluster [DBG] pgmap v933: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: audit 2023-11-17T07:17:48.798134+0000 mon.smithi097 (mon.0) 996 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: audit 2023-11-17T07:17:48.799393+0000 mon.smithi097 (mon.0) 997 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: audit 2023-11-17T07:17:48.806884+0000 mon.smithi097 (mon.0) 998 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: cluster 2023-11-17T07:17:48.807843+0000 mgr.smithi097.swvonl (mgr.14180) 1287 : cluster [DBG] pgmap v934: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: audit 2023-11-17T07:17:48.815729+0000 mon.smithi097 (mon.0) 999 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: audit 2023-11-17T07:17:48.830752+0000 mon.smithi097 (mon.0) 1000 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:49 smithi097 bash[24958]: cephadm 2023-11-17T07:17:48.834378+0000 mgr.smithi097.swvonl (mgr.14180) 1288 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.yutmgg on smithi193 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:50 smithi097 bash[24958]: cluster 2023-11-17T07:17:49.804072+0000 mon.smithi097 (mon.0) 1001 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:50 smithi097 bash[24958]: cluster 2023-11-17T07:17:49.804145+0000 mon.smithi097 (mon.0) 1002 : cluster [INF] Cluster is now healthy 2023-11-17T07:18:31.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:51 smithi097 bash[24958]: cluster 2023-11-17T07:17:50.808916+0000 mgr.smithi097.swvonl (mgr.14180) 1289 : cluster [DBG] pgmap v935: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:52 smithi097 bash[24958]: audit 2023-11-17T07:17:52.641988+0000 mon.smithi097 (mon.0) 1003 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.yutmgg"}]: dispatch 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: cephadm 2023-11-17T07:17:52.640776+0000 mgr.smithi097.swvonl (mgr.14180) 1290 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: cephadm 2023-11-17T07:17:52.641292+0000 mgr.smithi097.swvonl (mgr.14180) 1291 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: cephadm 2023-11-17T07:17:52.643738+0000 mgr.smithi097.swvonl (mgr.14180) 1292 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.yutmgg ... 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.449 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: cephadm 2023-11-17T07:17:52.649265+0000 mgr.smithi097.swvonl (mgr.14180) 1293 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ztkpag on smithi097 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:53 smithi097 bash[24958]: cluster 2023-11-17T07:17:52.809446+0000 mgr.smithi097.swvonl (mgr.14180) 1294 : cluster [DBG] pgmap v936: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:55 smithi097 bash[24958]: cluster 2023-11-17T07:17:54.810457+0000 mgr.smithi097.swvonl (mgr.14180) 1295 : cluster [DBG] pgmap v937: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:56 smithi097 bash[24958]: audit 2023-11-17T07:17:56.442616+0000 mon.smithi097 (mon.0) 1004 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ztkpag"}]: dispatch 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:56 smithi097 bash[24958]: audit 2023-11-17T07:17:56.546128+0000 mon.smithi097 (mon.0) 1005 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cephadm 2023-11-17T07:17:56.441774+0000 mgr.smithi097.swvonl (mgr.14180) 1296 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.451 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cephadm 2023-11-17T07:17:56.441997+0000 mgr.smithi097.swvonl (mgr.14180) 1297 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cephadm 2023-11-17T07:17:56.443819+0000 mgr.smithi097.swvonl (mgr.14180) 1298 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ztkpag ... 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cephadm 2023-11-17T07:17:56.445856+0000 mgr.smithi097.swvonl (mgr.14180) 1299 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cephadm 2023-11-17T07:17:56.447710+0000 mgr.smithi097.swvonl (mgr.14180) 1300 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cluster 2023-11-17T07:17:56.448941+0000 mgr.smithi097.swvonl (mgr.14180) 1301 : cluster [DBG] pgmap v938: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 201 B/s rd, 0 op/s 2023-11-17T07:18:31.453 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:57 smithi097 bash[24958]: cluster 2023-11-17T07:17:56.867875+0000 mon.smithi097 (mon.0) 1006 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:59 smithi097 bash[24958]: cluster 2023-11-17T07:17:58.449866+0000 mgr.smithi097.swvonl (mgr.14180) 1302 : cluster [DBG] pgmap v939: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 201 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:17:59 smithi097 bash[24958]: audit 2023-11-17T07:17:59.280697+0000 mon.smithi097 (mon.0) 1007 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:01 smithi097 bash[24958]: cluster 2023-11-17T07:18:00.451341+0000 mgr.smithi097.swvonl (mgr.14180) 1303 : cluster [DBG] pgmap v940: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:03 smithi097 bash[24958]: cluster 2023-11-17T07:18:02.452388+0000 mgr.smithi097.swvonl (mgr.14180) 1304 : cluster [DBG] pgmap v941: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:05 smithi097 bash[24958]: cluster 2023-11-17T07:18:04.453504+0000 mgr.smithi097.swvonl (mgr.14180) 1305 : cluster [DBG] pgmap v942: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:07 smithi097 bash[24958]: cluster 2023-11-17T07:18:06.454822+0000 mgr.smithi097.swvonl (mgr.14180) 1306 : cluster [DBG] pgmap v943: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 175 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:09 smithi097 bash[24958]: cluster 2023-11-17T07:18:08.455894+0000 mgr.smithi097.swvonl (mgr.14180) 1307 : cluster [DBG] pgmap v944: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:11 smithi097 bash[24958]: cluster 2023-11-17T07:18:10.457650+0000 mgr.smithi097.swvonl (mgr.14180) 1308 : cluster [DBG] pgmap v945: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:11 smithi097 bash[24958]: audit 2023-11-17T07:18:11.546620+0000 mon.smithi097 (mon.0) 1008 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:13 smithi097 bash[24958]: cluster 2023-11-17T07:18:12.458684+0000 mgr.smithi097.swvonl (mgr.14180) 1309 : cluster [DBG] pgmap v946: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:15 smithi097 bash[24958]: cluster 2023-11-17T07:18:14.459828+0000 mgr.smithi097.swvonl (mgr.14180) 1310 : cluster [DBG] pgmap v947: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:17 smithi097 bash[24958]: cluster 2023-11-17T07:18:16.461340+0000 mgr.smithi097.swvonl (mgr.14180) 1311 : cluster [DBG] pgmap v948: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:19 smithi097 bash[24958]: cluster 2023-11-17T07:18:18.462533+0000 mgr.smithi097.swvonl (mgr.14180) 1312 : cluster [DBG] pgmap v949: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:21 smithi097 bash[24958]: cluster 2023-11-17T07:18:20.464097+0000 mgr.smithi097.swvonl (mgr.14180) 1313 : cluster [DBG] pgmap v950: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:31.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:23 smithi097 bash[24958]: cluster 2023-11-17T07:18:22.465213+0000 mgr.smithi097.swvonl (mgr.14180) 1314 : cluster [DBG] pgmap v951: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:25 smithi097 bash[24958]: cluster 2023-11-17T07:18:24.466381+0000 mgr.smithi097.swvonl (mgr.14180) 1315 : cluster [DBG] pgmap v952: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:27 smithi097 bash[24958]: cluster 2023-11-17T07:18:26.467682+0000 mgr.smithi097.swvonl (mgr.14180) 1316 : cluster [DBG] pgmap v953: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:31.455 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:29 smithi097 bash[24958]: cluster 2023-11-17T07:18:28.468694+0000 mgr.smithi097.swvonl (mgr.14180) 1317 : cluster [DBG] pgmap v954: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:32.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:31 smithi097 bash[24958]: cluster 2023-11-17T07:18:30.470062+0000 mgr.smithi097.swvonl (mgr.14180) 1318 : cluster [DBG] pgmap v955: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:32.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:31 smithi193 bash[28559]: cluster 2023-11-17T07:18:30.470062+0000 mgr.smithi097.swvonl (mgr.14180) 1318 : cluster [DBG] pgmap v955: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:32.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:18:33.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:32 smithi097 bash[24958]: cluster 2023-11-17T07:18:32.470975+0000 mgr.smithi097.swvonl (mgr.14180) 1319 : cluster [DBG] pgmap v956: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:33.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:32 smithi193 bash[28559]: cluster 2023-11-17T07:18:32.470975+0000 mgr.smithi097.swvonl (mgr.14180) 1319 : cluster [DBG] pgmap v956: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:35.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:35 smithi193 bash[28559]: cluster 2023-11-17T07:18:34.472074+0000 mgr.smithi097.swvonl (mgr.14180) 1320 : cluster [DBG] pgmap v957: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:35.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:35 smithi097 bash[24958]: cluster 2023-11-17T07:18:34.472074+0000 mgr.smithi097.swvonl (mgr.14180) 1320 : cluster [DBG] pgmap v957: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:36.233 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:18:36.234 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:16:48.248823Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:52.643425Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\nDeploy daemon haproxy.nfs.foo.smithi193.yutmgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.443700Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\nDeploy daemon haproxy.nfs.foo.smithi097.ztkpag ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.445750Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:18:37.098 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:18:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:37 smithi097 bash[24958]: audit 2023-11-17T07:18:36.222814+0000 mgr.smithi097.swvonl (mgr.14180) 1321 : audit [DBG] from='client.15064 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:37 smithi097 bash[24958]: cluster 2023-11-17T07:18:36.473506+0000 mgr.smithi097.swvonl (mgr.14180) 1322 : cluster [DBG] pgmap v958: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:38.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:37 smithi193 bash[28559]: audit 2023-11-17T07:18:36.222814+0000 mgr.smithi097.swvonl (mgr.14180) 1321 : audit [DBG] from='client.15064 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:38.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:37 smithi193 bash[28559]: cluster 2023-11-17T07:18:36.473506+0000 mgr.smithi097.swvonl (mgr.14180) 1322 : cluster [DBG] pgmap v958: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:38.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:18:39.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:39 smithi097 bash[24958]: cluster 2023-11-17T07:18:38.474276+0000 mgr.smithi097.swvonl (mgr.14180) 1323 : cluster [DBG] pgmap v959: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:40.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:39 smithi193 bash[28559]: cluster 2023-11-17T07:18:38.474276+0000 mgr.smithi097.swvonl (mgr.14180) 1323 : cluster [DBG] pgmap v959: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:41.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:41 smithi097 bash[24958]: cluster 2023-11-17T07:18:40.475132+0000 mgr.smithi097.swvonl (mgr.14180) 1324 : cluster [DBG] pgmap v960: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:41.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:18:41.893 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:16:48.248823Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:52.643425Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\nDeploy daemon haproxy.nfs.foo.smithi193.yutmgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.443700Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\nDeploy daemon haproxy.nfs.foo.smithi097.ztkpag ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.445750Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:18:42.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:41 smithi193 bash[28559]: cluster 2023-11-17T07:18:40.475132+0000 mgr.smithi097.swvonl (mgr.14180) 1324 : cluster [DBG] pgmap v960: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:42.780 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:18:43.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:42 smithi097 bash[24958]: audit 2023-11-17T07:18:41.886859+0000 mgr.smithi097.swvonl (mgr.14180) 1325 : audit [DBG] from='client.15068 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:43.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:42 smithi193 bash[28559]: audit 2023-11-17T07:18:41.886859+0000 mgr.smithi097.swvonl (mgr.14180) 1325 : audit [DBG] from='client.15068 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:43.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:18:43.984 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:43 smithi097 bash[24958]: cluster 2023-11-17T07:18:42.476120+0000 mgr.smithi097.swvonl (mgr.14180) 1326 : cluster [DBG] pgmap v961: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:44.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:43 smithi193 bash[28559]: cluster 2023-11-17T07:18:42.476120+0000 mgr.smithi097.swvonl (mgr.14180) 1326 : cluster [DBG] pgmap v961: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:45.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:45 smithi097 bash[24958]: cluster 2023-11-17T07:18:44.476769+0000 mgr.smithi097.swvonl (mgr.14180) 1327 : cluster [DBG] pgmap v962: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:46.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:45 smithi193 bash[28559]: cluster 2023-11-17T07:18:44.476769+0000 mgr.smithi097.swvonl (mgr.14180) 1327 : cluster [DBG] pgmap v962: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:47.606 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:18:47.606 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:16:48.248823Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:52.643425Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\nDeploy daemon haproxy.nfs.foo.smithi193.yutmgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.443700Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\nDeploy daemon haproxy.nfs.foo.smithi097.ztkpag ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.445750Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:18:48.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:47 smithi097 bash[24958]: cluster 2023-11-17T07:18:46.478170+0000 mgr.smithi097.swvonl (mgr.14180) 1328 : cluster [DBG] pgmap v963: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:48.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:47 smithi193 bash[28559]: cluster 2023-11-17T07:18:46.478170+0000 mgr.smithi097.swvonl (mgr.14180) 1328 : cluster [DBG] pgmap v963: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:48.534 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:18:51.957 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:48 smithi193 bash[28559]: audit 2023-11-17T07:18:47.595568+0000 mgr.smithi097.swvonl (mgr.14180) 1329 : audit [DBG] from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:51.957 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:49 smithi193 bash[28559]: cluster 2023-11-17T07:18:48.478833+0000 mgr.smithi097.swvonl (mgr.14180) 1330 : cluster [DBG] pgmap v964: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:51.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:48 smithi097 bash[24958]: audit 2023-11-17T07:18:47.595568+0000 mgr.smithi097.swvonl (mgr.14180) 1329 : audit [DBG] from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:51.957 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:49 smithi097 bash[24958]: cluster 2023-11-17T07:18:48.478833+0000 mgr.smithi097.swvonl (mgr.14180) 1330 : cluster [DBG] pgmap v964: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:52.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:51 smithi097 bash[24958]: cluster 2023-11-17T07:18:50.480186+0000 mgr.smithi097.swvonl (mgr.14180) 1331 : cluster [DBG] pgmap v965: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:52.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:51 smithi193 bash[28559]: cluster 2023-11-17T07:18:50.480186+0000 mgr.smithi097.swvonl (mgr.14180) 1331 : cluster [DBG] pgmap v965: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:18:52.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:18:54.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:53 smithi097 bash[24958]: cluster 2023-11-17T07:18:52.481173+0000 mgr.smithi097.swvonl (mgr.14180) 1332 : cluster [DBG] pgmap v966: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:54.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:53 smithi193 bash[28559]: cluster 2023-11-17T07:18:52.481173+0000 mgr.smithi097.swvonl (mgr.14180) 1332 : cluster [DBG] pgmap v966: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:56.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:55 smithi097 bash[24958]: cluster 2023-11-17T07:18:54.482440+0000 mgr.smithi097.swvonl (mgr.14180) 1333 : cluster [DBG] pgmap v967: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:56.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:55 smithi193 bash[28559]: cluster 2023-11-17T07:18:54.482440+0000 mgr.smithi097.swvonl (mgr.14180) 1333 : cluster [DBG] pgmap v967: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:56.551 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:18:56.551 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:16:48.248823Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:52.643425Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\nDeploy daemon haproxy.nfs.foo.smithi193.yutmgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.443700Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\nDeploy daemon haproxy.nfs.foo.smithi097.ztkpag ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.445750Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:18:57.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:56 smithi097 bash[24958]: audit 2023-11-17T07:18:56.452199+0000 mon.smithi097 (mon.0) 1009 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:57.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:56 smithi193 bash[28559]: audit 2023-11-17T07:18:56.452199+0000 mon.smithi097 (mon.0) 1009 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:18:57.431 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:18:58.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:57 smithi097 bash[24958]: cluster 2023-11-17T07:18:56.483336+0000 mgr.smithi097.swvonl (mgr.14180) 1334 : cluster [DBG] pgmap v968: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:58.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:57 smithi097 bash[24958]: audit 2023-11-17T07:18:56.541492+0000 mgr.smithi097.swvonl (mgr.14180) 1335 : audit [DBG] from='client.15076 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:57 smithi193 bash[28559]: cluster 2023-11-17T07:18:56.483336+0000 mgr.smithi097.swvonl (mgr.14180) 1334 : cluster [DBG] pgmap v968: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:18:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:57 smithi193 bash[28559]: audit 2023-11-17T07:18:56.541492+0000 mgr.smithi097.swvonl (mgr.14180) 1335 : audit [DBG] from='client.15076 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:18:58.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:00.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:18:59 smithi097 bash[24958]: cluster 2023-11-17T07:18:58.484362+0000 mgr.smithi097.swvonl (mgr.14180) 1336 : cluster [DBG] pgmap v969: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:00.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:18:59 smithi193 bash[28559]: cluster 2023-11-17T07:18:58.484362+0000 mgr.smithi097.swvonl (mgr.14180) 1336 : cluster [DBG] pgmap v969: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:02.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:01 smithi097 bash[24958]: cluster 2023-11-17T07:19:00.485781+0000 mgr.smithi097.swvonl (mgr.14180) 1337 : cluster [DBG] pgmap v970: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:05.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:01 smithi193 bash[28559]: cluster 2023-11-17T07:19:00.485781+0000 mgr.smithi097.swvonl (mgr.14180) 1337 : cluster [DBG] pgmap v970: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:05.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.151131+0000 mgr.smithi097.swvonl (mgr.14180) 1338 : audit [DBG] from='client.15080 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:05.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.198820+0000 mon.smithi097 (mon.0) 1010 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: cluster 2023-11-17T07:19:02.486779+0000 mgr.smithi097.swvonl (mgr.14180) 1339 : cluster [DBG] pgmap v971: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.659294+0000 mon.smithi097 (mon.0) 1011 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.941059+0000 mon.smithi097 (mon.0) 1012 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.942316+0000 mon.smithi097 (mon.0) 1013 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.949710+0000 mon.smithi097 (mon.0) 1014 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: cluster 2023-11-17T07:19:02.950421+0000 mgr.smithi097.swvonl (mgr.14180) 1340 : cluster [DBG] pgmap v972: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.956172+0000 mon.smithi097 (mon.0) 1015 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: audit 2023-11-17T07:19:02.967092+0000 mon.smithi097 (mon.0) 1016 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:03 smithi193 bash[28559]: cephadm 2023-11-17T07:19:02.970240+0000 mgr.smithi097.swvonl (mgr.14180) 1341 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ueihog on smithi193 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:04 smithi193 bash[28559]: cluster 2023-11-17T07:19:03.197709+0000 mon.smithi097 (mon.0) 1017 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:19:05.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:04 smithi193 bash[28559]: cluster 2023-11-17T07:19:03.197798+0000 mon.smithi097 (mon.0) 1018 : cluster [INF] Cluster is now healthy 2023-11-17T07:19:05.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:05 smithi193 bash[28559]: cluster 2023-11-17T07:19:04.951465+0000 mgr.smithi097.swvonl (mgr.14180) 1342 : cluster [DBG] pgmap v973: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:05.990 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:05.990 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:08:41.841330Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:08:41.372521Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:08:41.841741Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:16:48.248823Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.faqnnl on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:52.643425Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.yutmgg on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-yutmgg\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.yutmgg\nDeploy daemon haproxy.nfs.foo.smithi193.yutmgg ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.443700Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ztkpag on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ztkpag\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ztkpag\nDeploy daemon haproxy.nfs.foo.smithi097.ztkpag ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:17:56.445750Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.crytdi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:08:41.375110Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:08:41.372962Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:08:41.373247Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:08:41.375459Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:08:41.373514Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:08:41.373757Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:08:41.842145Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:05.991 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:05.991 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.151131+0000 mgr.smithi097.swvonl (mgr.14180) 1338 : audit [DBG] from='client.15080 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:05.991 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.198820+0000 mon.smithi097 (mon.0) 1010 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: cluster 2023-11-17T07:19:02.486779+0000 mgr.smithi097.swvonl (mgr.14180) 1339 : cluster [DBG] pgmap v971: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.659294+0000 mon.smithi097 (mon.0) 1011 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.941059+0000 mon.smithi097 (mon.0) 1012 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.942316+0000 mon.smithi097 (mon.0) 1013 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.949710+0000 mon.smithi097 (mon.0) 1014 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: cluster 2023-11-17T07:19:02.950421+0000 mgr.smithi097.swvonl (mgr.14180) 1340 : cluster [DBG] pgmap v972: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.956172+0000 mon.smithi097 (mon.0) 1015 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: audit 2023-11-17T07:19:02.967092+0000 mon.smithi097 (mon.0) 1016 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:03 smithi097 bash[24958]: cephadm 2023-11-17T07:19:02.970240+0000 mgr.smithi097.swvonl (mgr.14180) 1341 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ueihog on smithi193 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:04 smithi097 bash[24958]: cluster 2023-11-17T07:19:03.197709+0000 mon.smithi097 (mon.0) 1017 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:04 smithi097 bash[24958]: cluster 2023-11-17T07:19:03.197798+0000 mon.smithi097 (mon.0) 1018 : cluster [INF] Cluster is now healthy 2023-11-17T07:19:05.993 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:05 smithi097 bash[24958]: cluster 2023-11-17T07:19:04.951465+0000 mgr.smithi097.swvonl (mgr.14180) 1342 : cluster [DBG] pgmap v973: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:06.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:08.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: cephadm 2023-11-17T07:19:06.822406+0000 mgr.smithi097.swvonl (mgr.14180) 1343 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: cephadm 2023-11-17T07:19:06.822961+0000 mgr.smithi097.swvonl (mgr.14180) 1344 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: audit 2023-11-17T07:19:06.823662+0000 mon.smithi097 (mon.0) 1019 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ueihog"}]: dispatch 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: cephadm 2023-11-17T07:19:06.825298+0000 mgr.smithi097.swvonl (mgr.14180) 1345 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: cephadm 2023-11-17T07:19:06.829492+0000 mgr.smithi097.swvonl (mgr.14180) 1346 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.txunvo on smithi097 2023-11-17T07:19:08.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:07 smithi097 bash[24958]: cluster 2023-11-17T07:19:06.952249+0000 mgr.smithi097.swvonl (mgr.14180) 1347 : cluster [DBG] pgmap v974: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:08.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: cephadm 2023-11-17T07:19:06.822406+0000 mgr.smithi097.swvonl (mgr.14180) 1343 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:19:08.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: cephadm 2023-11-17T07:19:06.822961+0000 mgr.smithi097.swvonl (mgr.14180) 1344 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: audit 2023-11-17T07:19:06.823662+0000 mon.smithi097 (mon.0) 1019 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ueihog"}]: dispatch 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: cephadm 2023-11-17T07:19:06.825298+0000 mgr.smithi097.swvonl (mgr.14180) 1345 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:08.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ueihog ... 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: cephadm 2023-11-17T07:19:06.829492+0000 mgr.smithi097.swvonl (mgr.14180) 1346 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.txunvo on smithi097 2023-11-17T07:19:08.063 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:07 smithi193 bash[28559]: cluster 2023-11-17T07:19:06.952249+0000 mgr.smithi097.swvonl (mgr.14180) 1347 : cluster [DBG] pgmap v974: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:09.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:09 smithi097 bash[24958]: cluster 2023-11-17T07:19:08.952968+0000 mgr.smithi097.swvonl (mgr.14180) 1348 : cluster [DBG] pgmap v975: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:09.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:09 smithi193 bash[28559]: cluster 2023-11-17T07:19:08.952968+0000 mgr.smithi097.swvonl (mgr.14180) 1348 : cluster [DBG] pgmap v975: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:19:10.600 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:10.600 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:10.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:10 smithi097 bash[24958]: audit 2023-11-17T07:19:10.577192+0000 mon.smithi097 (mon.0) 1020 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.txunvo"}]: dispatch 2023-11-17T07:19:11.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:10 smithi193 bash[28559]: audit 2023-11-17T07:19:10.577192+0000 mon.smithi097 (mon.0) 1020 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.txunvo"}]: dispatch 2023-11-17T07:19:11.529 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cephadm 2023-11-17T07:19:10.576399+0000 mgr.smithi097.swvonl (mgr.14180) 1349 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cephadm 2023-11-17T07:19:10.576606+0000 mgr.smithi097.swvonl (mgr.14180) 1350 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cephadm 2023-11-17T07:19:10.578369+0000 mgr.smithi097.swvonl (mgr.14180) 1351 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cephadm 2023-11-17T07:19:10.580316+0000 mgr.smithi097.swvonl (mgr.14180) 1352 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cephadm 2023-11-17T07:19:10.581949+0000 mgr.smithi097.swvonl (mgr.14180) 1353 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cluster 2023-11-17T07:19:10.583290+0000 mgr.smithi097.swvonl (mgr.14180) 1354 : cluster [DBG] pgmap v976: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:19:14.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: audit 2023-11-17T07:19:10.590973+0000 mgr.smithi097.swvonl (mgr.14180) 1355 : audit [DBG] from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:11 smithi193 bash[28559]: cluster 2023-11-17T07:19:10.635164+0000 mon.smithi097 (mon.0) 1021 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:13 smithi193 bash[28559]: cluster 2023-11-17T07:19:12.584241+0000 mgr.smithi097.swvonl (mgr.14180) 1356 : cluster [DBG] pgmap v977: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cephadm 2023-11-17T07:19:10.576399+0000 mgr.smithi097.swvonl (mgr.14180) 1349 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.791 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.792 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cephadm 2023-11-17T07:19:10.576606+0000 mgr.smithi097.swvonl (mgr.14180) 1350 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cephadm 2023-11-17T07:19:10.578369+0000 mgr.smithi097.swvonl (mgr.14180) 1351 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:19:14.793 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.txunvo ... 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cephadm 2023-11-17T07:19:10.580316+0000 mgr.smithi097.swvonl (mgr.14180) 1352 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cephadm 2023-11-17T07:19:10.581949+0000 mgr.smithi097.swvonl (mgr.14180) 1353 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cluster 2023-11-17T07:19:10.583290+0000 mgr.smithi097.swvonl (mgr.14180) 1354 : cluster [DBG] pgmap v976: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: audit 2023-11-17T07:19:10.590973+0000 mgr.smithi097.swvonl (mgr.14180) 1355 : audit [DBG] from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:11 smithi097 bash[24958]: cluster 2023-11-17T07:19:10.635164+0000 mon.smithi097 (mon.0) 1021 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:19:14.794 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:13 smithi097 bash[24958]: cluster 2023-11-17T07:19:12.584241+0000 mgr.smithi097.swvonl (mgr.14180) 1356 : cluster [DBG] pgmap v977: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:19:15.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:15 smithi193 bash[28559]: audit 2023-11-17T07:19:14.337756+0000 mon.smithi097 (mon.0) 1022 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:15.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:15 smithi193 bash[28559]: cluster 2023-11-17T07:19:14.585308+0000 mgr.smithi097.swvonl (mgr.14180) 1357 : cluster [DBG] pgmap v978: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:15.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:15.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:15 smithi097 bash[24958]: audit 2023-11-17T07:19:14.337756+0000 mon.smithi097 (mon.0) 1022 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:19:15.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:15 smithi097 bash[24958]: cluster 2023-11-17T07:19:14.585308+0000 mgr.smithi097.swvonl (mgr.14180) 1357 : cluster [DBG] pgmap v978: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:17.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:17 smithi097 bash[24958]: cluster 2023-11-17T07:19:16.586569+0000 mgr.smithi097.swvonl (mgr.14180) 1358 : cluster [DBG] pgmap v979: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:18.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:17 smithi193 bash[28559]: cluster 2023-11-17T07:19:16.586569+0000 mgr.smithi097.swvonl (mgr.14180) 1358 : cluster [DBG] pgmap v979: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:19.535 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:19.535 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:19.843 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:19 smithi097 bash[24958]: cluster 2023-11-17T07:19:18.587561+0000 mgr.smithi097.swvonl (mgr.14180) 1359 : cluster [DBG] pgmap v980: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:20.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:19 smithi193 bash[28559]: cluster 2023-11-17T07:19:18.587561+0000 mgr.smithi097.swvonl (mgr.14180) 1359 : cluster [DBG] pgmap v980: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:19:20.419 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:20.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:20 smithi097 bash[24958]: audit 2023-11-17T07:19:19.525166+0000 mgr.smithi097.swvonl (mgr.14180) 1360 : audit [DBG] from='client.15088 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:21.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:20 smithi193 bash[28559]: audit 2023-11-17T07:19:19.525166+0000 mgr.smithi097.swvonl (mgr.14180) 1360 : audit [DBG] from='client.15088 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:21.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:21.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:21 smithi097 bash[24958]: cluster 2023-11-17T07:19:20.588930+0000 mgr.smithi097.swvonl (mgr.14180) 1361 : cluster [DBG] pgmap v981: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 352 B/s rd, 0 op/s 2023-11-17T07:19:22.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:21 smithi193 bash[28559]: cluster 2023-11-17T07:19:20.588930+0000 mgr.smithi097.swvonl (mgr.14180) 1361 : cluster [DBG] pgmap v981: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 352 B/s rd, 0 op/s 2023-11-17T07:19:24.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:23 smithi097 bash[24958]: cluster 2023-11-17T07:19:22.589930+0000 mgr.smithi097.swvonl (mgr.14180) 1362 : cluster [DBG] pgmap v982: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:24.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:23 smithi193 bash[28559]: cluster 2023-11-17T07:19:22.589930+0000 mgr.smithi097.swvonl (mgr.14180) 1362 : cluster [DBG] pgmap v982: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:25.396 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:25.397 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:25.701 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:25 smithi097 bash[24958]: cluster 2023-11-17T07:19:24.591029+0000 mgr.smithi097.swvonl (mgr.14180) 1363 : cluster [DBG] pgmap v983: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:26.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:25 smithi193 bash[28559]: cluster 2023-11-17T07:19:24.591029+0000 mgr.smithi097.swvonl (mgr.14180) 1363 : cluster [DBG] pgmap v983: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:26.285 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:27.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:26 smithi097 bash[24958]: audit 2023-11-17T07:19:25.385059+0000 mgr.smithi097.swvonl (mgr.14180) 1364 : audit [DBG] from='client.15092 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:27.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:26 smithi193 bash[28559]: audit 2023-11-17T07:19:25.385059+0000 mgr.smithi097.swvonl (mgr.14180) 1364 : audit [DBG] from='client.15092 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:27.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:27.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:27 smithi097 bash[24958]: cluster 2023-11-17T07:19:26.592349+0000 mgr.smithi097.swvonl (mgr.14180) 1365 : cluster [DBG] pgmap v984: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:28.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:27 smithi193 bash[28559]: cluster 2023-11-17T07:19:26.592349+0000 mgr.smithi097.swvonl (mgr.14180) 1365 : cluster [DBG] pgmap v984: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:29.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:29 smithi097 bash[24958]: cluster 2023-11-17T07:19:28.593313+0000 mgr.smithi097.swvonl (mgr.14180) 1366 : cluster [DBG] pgmap v985: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:29 smithi193 bash[28559]: cluster 2023-11-17T07:19:28.593313+0000 mgr.smithi097.swvonl (mgr.14180) 1366 : cluster [DBG] pgmap v985: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:30.988 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:30.989 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:31.832 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:32.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:31 smithi097 bash[24958]: cluster 2023-11-17T07:19:30.594810+0000 mgr.smithi097.swvonl (mgr.14180) 1367 : cluster [DBG] pgmap v986: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:32.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:31 smithi097 bash[24958]: audit 2023-11-17T07:19:30.978500+0000 mgr.smithi097.swvonl (mgr.14180) 1368 : audit [DBG] from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:32.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:31 smithi193 bash[28559]: cluster 2023-11-17T07:19:30.594810+0000 mgr.smithi097.swvonl (mgr.14180) 1367 : cluster [DBG] pgmap v986: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:32.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:31 smithi193 bash[28559]: audit 2023-11-17T07:19:30.978500+0000 mgr.smithi097.swvonl (mgr.14180) 1368 : audit [DBG] from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:32.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:33 smithi097 bash[24958]: cluster 2023-11-17T07:19:32.595797+0000 mgr.smithi097.swvonl (mgr.14180) 1369 : cluster [DBG] pgmap v987: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:34.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:33 smithi193 bash[28559]: cluster 2023-11-17T07:19:32.595797+0000 mgr.smithi097.swvonl (mgr.14180) 1369 : cluster [DBG] pgmap v987: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:35.992 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:35 smithi097 bash[24958]: cluster 2023-11-17T07:19:34.596889+0000 mgr.smithi097.swvonl (mgr.14180) 1370 : cluster [DBG] pgmap v988: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:35 smithi193 bash[28559]: cluster 2023-11-17T07:19:34.596889+0000 mgr.smithi097.swvonl (mgr.14180) 1370 : cluster [DBG] pgmap v988: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:37 smithi193 bash[28559]: audit 2023-11-17T07:19:36.340356+0000 mgr.smithi097.swvonl (mgr.14180) 1371 : audit [DBG] from='client.15100 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:43.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:37 smithi193 bash[28559]: cluster 2023-11-17T07:19:36.598144+0000 mgr.smithi097.swvonl (mgr.14180) 1372 : cluster [DBG] pgmap v989: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:39 smithi193 bash[28559]: cluster 2023-11-17T07:19:38.599282+0000 mgr.smithi097.swvonl (mgr.14180) 1373 : cluster [DBG] pgmap v990: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:41 smithi193 bash[28559]: cluster 2023-11-17T07:19:40.600651+0000 mgr.smithi097.swvonl (mgr.14180) 1374 : cluster [DBG] pgmap v991: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:43.303 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:43.303 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:43.304 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:43.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:37 smithi097 bash[24958]: audit 2023-11-17T07:19:36.340356+0000 mgr.smithi097.swvonl (mgr.14180) 1371 : audit [DBG] from='client.15100 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:43.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:37 smithi097 bash[24958]: cluster 2023-11-17T07:19:36.598144+0000 mgr.smithi097.swvonl (mgr.14180) 1372 : cluster [DBG] pgmap v989: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:39 smithi097 bash[24958]: cluster 2023-11-17T07:19:38.599282+0000 mgr.smithi097.swvonl (mgr.14180) 1373 : cluster [DBG] pgmap v990: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:43.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:41 smithi097 bash[24958]: cluster 2023-11-17T07:19:40.600651+0000 mgr.smithi097.swvonl (mgr.14180) 1374 : cluster [DBG] pgmap v991: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:44.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:43 smithi097 bash[24958]: cluster 2023-11-17T07:19:42.601697+0000 mgr.smithi097.swvonl (mgr.14180) 1375 : cluster [DBG] pgmap v992: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:44.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:43 smithi193 bash[28559]: cluster 2023-11-17T07:19:42.601697+0000 mgr.smithi097.swvonl (mgr.14180) 1375 : cluster [DBG] pgmap v992: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:44.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:46.023 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:45 smithi097 bash[24958]: cluster 2023-11-17T07:19:44.602950+0000 mgr.smithi097.swvonl (mgr.14180) 1376 : cluster [DBG] pgmap v993: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:46.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:45 smithi193 bash[28559]: cluster 2023-11-17T07:19:44.602950+0000 mgr.smithi097.swvonl (mgr.14180) 1376 : cluster [DBG] pgmap v993: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:47.917 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:47.917 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:48.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:47 smithi097 bash[24958]: cluster 2023-11-17T07:19:46.604264+0000 mgr.smithi097.swvonl (mgr.14180) 1377 : cluster [DBG] pgmap v994: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:48.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:47 smithi193 bash[28559]: cluster 2023-11-17T07:19:46.604264+0000 mgr.smithi097.swvonl (mgr.14180) 1377 : cluster [DBG] pgmap v994: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:48.759 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:49.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:48 smithi097 bash[24958]: audit 2023-11-17T07:19:47.906343+0000 mgr.smithi097.swvonl (mgr.14180) 1378 : audit [DBG] from='client.15104 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:49.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:48 smithi193 bash[28559]: audit 2023-11-17T07:19:47.906343+0000 mgr.smithi097.swvonl (mgr.14180) 1378 : audit [DBG] from='client.15104 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:49.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:50.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:49 smithi097 bash[24958]: cluster 2023-11-17T07:19:48.605377+0000 mgr.smithi097.swvonl (mgr.14180) 1379 : cluster [DBG] pgmap v995: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:50.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:49 smithi193 bash[28559]: cluster 2023-11-17T07:19:48.605377+0000 mgr.smithi097.swvonl (mgr.14180) 1379 : cluster [DBG] pgmap v995: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:52.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:51 smithi097 bash[24958]: cluster 2023-11-17T07:19:50.606746+0000 mgr.smithi097.swvonl (mgr.14180) 1380 : cluster [DBG] pgmap v996: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:52.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:51 smithi193 bash[28559]: cluster 2023-11-17T07:19:50.606746+0000 mgr.smithi097.swvonl (mgr.14180) 1380 : cluster [DBG] pgmap v996: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:19:53.649 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:53.649 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:54.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:53 smithi097 bash[24958]: cluster 2023-11-17T07:19:52.607671+0000 mgr.smithi097.swvonl (mgr.14180) 1381 : cluster [DBG] pgmap v997: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:54.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:53 smithi193 bash[28559]: cluster 2023-11-17T07:19:52.607671+0000 mgr.smithi097.swvonl (mgr.14180) 1381 : cluster [DBG] pgmap v997: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:54.487 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:19:55.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:54 smithi097 bash[24958]: audit 2023-11-17T07:19:53.643181+0000 mgr.smithi097.swvonl (mgr.14180) 1382 : audit [DBG] from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:55.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:54 smithi193 bash[28559]: audit 2023-11-17T07:19:53.643181+0000 mgr.smithi097.swvonl (mgr.14180) 1382 : audit [DBG] from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:19:55.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:19:56.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:55 smithi097 bash[24958]: cluster 2023-11-17T07:19:54.608716+0000 mgr.smithi097.swvonl (mgr.14180) 1383 : cluster [DBG] pgmap v998: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:56.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:55 smithi193 bash[28559]: cluster 2023-11-17T07:19:54.608716+0000 mgr.smithi097.swvonl (mgr.14180) 1383 : cluster [DBG] pgmap v998: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:58.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:57 smithi097 bash[24958]: cluster 2023-11-17T07:19:56.610071+0000 mgr.smithi097.swvonl (mgr.14180) 1384 : cluster [DBG] pgmap v999: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:57 smithi193 bash[28559]: cluster 2023-11-17T07:19:56.610071+0000 mgr.smithi097.swvonl (mgr.14180) 1384 : cluster [DBG] pgmap v999: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:19:59.499 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:19:59.500 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:19:59.832 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:19:59 smithi097 bash[24958]: cluster 2023-11-17T07:19:58.611254+0000 mgr.smithi097.swvonl (mgr.14180) 1385 : cluster [DBG] pgmap v1000: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:00.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:19:59 smithi193 bash[28559]: cluster 2023-11-17T07:19:58.611254+0000 mgr.smithi097.swvonl (mgr.14180) 1385 : cluster [DBG] pgmap v1000: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:00.387 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:01.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:00 smithi193 bash[28559]: audit 2023-11-17T07:19:59.489144+0000 mgr.smithi097.swvonl (mgr.14180) 1386 : audit [DBG] from='client.15112 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:01.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:00 smithi193 bash[28559]: cluster 2023-11-17T07:20:00.000116+0000 mon.smithi097 (mon.0) 1023 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:20:01.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:00 smithi097 bash[24958]: audit 2023-11-17T07:19:59.489144+0000 mgr.smithi097.swvonl (mgr.14180) 1386 : audit [DBG] from='client.15112 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:01.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:00 smithi097 bash[24958]: cluster 2023-11-17T07:20:00.000116+0000 mon.smithi097 (mon.0) 1023 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:20:01.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:02.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:01 smithi097 bash[24958]: cluster 2023-11-17T07:20:00.612850+0000 mgr.smithi097.swvonl (mgr.14180) 1387 : cluster [DBG] pgmap v1001: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:02.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:01 smithi193 bash[28559]: cluster 2023-11-17T07:20:00.612850+0000 mgr.smithi097.swvonl (mgr.14180) 1387 : cluster [DBG] pgmap v1001: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:04.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:03 smithi097 bash[24958]: cluster 2023-11-17T07:20:02.613461+0000 mgr.smithi097.swvonl (mgr.14180) 1388 : cluster [DBG] pgmap v1002: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:04.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:03 smithi193 bash[28559]: cluster 2023-11-17T07:20:02.613461+0000 mgr.smithi097.swvonl (mgr.14180) 1388 : cluster [DBG] pgmap v1002: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:05.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:05.220 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:06.084 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:05 smithi097 bash[24958]: cluster 2023-11-17T07:20:04.614546+0000 mgr.smithi097.swvonl (mgr.14180) 1389 : cluster [DBG] pgmap v1003: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:06.085 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:06.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:05 smithi193 bash[28559]: cluster 2023-11-17T07:20:04.614546+0000 mgr.smithi097.swvonl (mgr.14180) 1389 : cluster [DBG] pgmap v1003: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:07.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:07.099 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:06 smithi097 bash[24958]: audit 2023-11-17T07:20:05.207699+0000 mgr.smithi097.swvonl (mgr.14180) 1390 : audit [DBG] from='client.15116 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:07.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:06 smithi193 bash[28559]: audit 2023-11-17T07:20:05.207699+0000 mgr.smithi097.swvonl (mgr.14180) 1390 : audit [DBG] from='client.15116 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:08.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:07 smithi097 bash[24958]: cluster 2023-11-17T07:20:06.615851+0000 mgr.smithi097.swvonl (mgr.14180) 1391 : cluster [DBG] pgmap v1004: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:08.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:07 smithi193 bash[28559]: cluster 2023-11-17T07:20:06.615851+0000 mgr.smithi097.swvonl (mgr.14180) 1391 : cluster [DBG] pgmap v1004: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:10.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:09 smithi097 bash[24958]: cluster 2023-11-17T07:20:08.616936+0000 mgr.smithi097.swvonl (mgr.14180) 1392 : cluster [DBG] pgmap v1005: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:10.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:09 smithi193 bash[28559]: cluster 2023-11-17T07:20:08.616936+0000 mgr.smithi097.swvonl (mgr.14180) 1392 : cluster [DBG] pgmap v1005: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:10.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:10.548 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:17:56.447592Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.vhjpla on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:10 smithi097 bash[24958]: audit 2023-11-17T07:20:10.585535+0000 mon.smithi097 (mon.0) 1024 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:20:11.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:10 smithi193 bash[28559]: audit 2023-11-17T07:20:10.585535+0000 mon.smithi097 (mon.0) 1024 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:20:11.382 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:10.537249+0000 mgr.smithi097.swvonl (mgr.14180) 1393 : audit [DBG] from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: cluster 2023-11-17T07:20:10.617995+0000 mgr.smithi097.swvonl (mgr.14180) 1394 : cluster [DBG] pgmap v1006: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:10.863510+0000 mon.smithi097 (mon.0) 1025 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:11.132472+0000 mon.smithi097 (mon.0) 1026 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:11.133975+0000 mon.smithi097 (mon.0) 1027 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:20:12.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:11.143696+0000 mon.smithi097 (mon.0) 1028 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:11.153114+0000 mon.smithi097 (mon.0) 1029 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:20:12.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:11 smithi097 bash[24958]: audit 2023-11-17T07:20:11.165980+0000 mon.smithi097 (mon.0) 1030 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:10.537249+0000 mgr.smithi097.swvonl (mgr.14180) 1393 : audit [DBG] from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: cluster 2023-11-17T07:20:10.617995+0000 mgr.smithi097.swvonl (mgr.14180) 1394 : cluster [DBG] pgmap v1006: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:10.863510+0000 mon.smithi097 (mon.0) 1025 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:11.132472+0000 mon.smithi097 (mon.0) 1026 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:11.133975+0000 mon.smithi097 (mon.0) 1027 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:11.143696+0000 mon.smithi097 (mon.0) 1028 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:11.153114+0000 mon.smithi097 (mon.0) 1029 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:20:12.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:11 smithi193 bash[28559]: audit 2023-11-17T07:20:11.165980+0000 mon.smithi097 (mon.0) 1030 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:12.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:12 smithi097 bash[24958]: cluster 2023-11-17T07:20:11.144572+0000 mgr.smithi097.swvonl (mgr.14180) 1395 : cluster [DBG] pgmap v1007: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:12 smithi097 bash[24958]: cephadm 2023-11-17T07:20:11.169018+0000 mgr.smithi097.swvonl (mgr.14180) 1396 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ppovlp on smithi193 2023-11-17T07:20:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:12 smithi097 bash[24958]: cluster 2023-11-17T07:20:12.140265+0000 mon.smithi097 (mon.0) 1031 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:20:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:12 smithi097 bash[24958]: cluster 2023-11-17T07:20:12.140338+0000 mon.smithi097 (mon.0) 1032 : cluster [INF] Cluster is now healthy 2023-11-17T07:20:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:12 smithi193 bash[28559]: cluster 2023-11-17T07:20:11.144572+0000 mgr.smithi097.swvonl (mgr.14180) 1395 : cluster [DBG] pgmap v1007: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:12 smithi193 bash[28559]: cephadm 2023-11-17T07:20:11.169018+0000 mgr.smithi097.swvonl (mgr.14180) 1396 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ppovlp on smithi193 2023-11-17T07:20:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:12 smithi193 bash[28559]: cluster 2023-11-17T07:20:12.140265+0000 mon.smithi097 (mon.0) 1031 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:20:13.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:12 smithi193 bash[28559]: cluster 2023-11-17T07:20:12.140338+0000 mon.smithi097 (mon.0) 1032 : cluster [INF] Cluster is now healthy 2023-11-17T07:20:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:14 smithi193 bash[28559]: cluster 2023-11-17T07:20:13.145081+0000 mgr.smithi097.swvonl (mgr.14180) 1397 : cluster [DBG] pgmap v1008: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:15.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:14 smithi097 bash[24958]: cluster 2023-11-17T07:20:13.145081+0000 mgr.smithi097.swvonl (mgr.14180) 1397 : cluster [DBG] pgmap v1008: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:16.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:16.243 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:06.825083Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ueihog on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ueihog\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ueihog\nDeploy daemon haproxy.nfs.foo.smithi193.ueihog ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.578212Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.txunvo on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-txunvo\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.txunvo\nDeploy daemon haproxy.nfs.foo.smithi097.txunvo ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:19:10.580225Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.kdvmsq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:16.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:15 smithi097 bash[24958]: audit 2023-11-17T07:20:15.361495+0000 mon.smithi097 (mon.0) 1033 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ppovlp"}]: dispatch 2023-11-17T07:20:16.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:15 smithi193 bash[28559]: audit 2023-11-17T07:20:15.361495+0000 mon.smithi097 (mon.0) 1033 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ppovlp"}]: dispatch 2023-11-17T07:20:17.175 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: cluster 2023-11-17T07:20:15.146215+0000 mgr.smithi097.swvonl (mgr.14180) 1398 : cluster [DBG] pgmap v1009: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: cephadm 2023-11-17T07:20:15.360426+0000 mgr.smithi097.swvonl (mgr.14180) 1399 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:20:17.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: cephadm 2023-11-17T07:20:15.360859+0000 mgr.smithi097.swvonl (mgr.14180) 1400 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: cephadm 2023-11-17T07:20:15.363076+0000 mgr.smithi097.swvonl (mgr.14180) 1401 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.178 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.179 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:16 smithi097 bash[24958]: cephadm 2023-11-17T07:20:15.365566+0000 mgr.smithi097.swvonl (mgr.14180) 1402 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.nythfr on smithi097 2023-11-17T07:20:17.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: cluster 2023-11-17T07:20:15.146215+0000 mgr.smithi097.swvonl (mgr.14180) 1398 : cluster [DBG] pgmap v1009: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:17.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: cephadm 2023-11-17T07:20:15.360426+0000 mgr.smithi097.swvonl (mgr.14180) 1399 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: cephadm 2023-11-17T07:20:15.360859+0000 mgr.smithi097.swvonl (mgr.14180) 1400 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: cephadm 2023-11-17T07:20:15.363076+0000 mgr.smithi097.swvonl (mgr.14180) 1401 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp 2023-11-17T07:20:17.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ppovlp ... 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:17.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:16 smithi193 bash[28559]: cephadm 2023-11-17T07:20:15.365566+0000 mgr.smithi097.swvonl (mgr.14180) 1402 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.nythfr on smithi097 2023-11-17T07:20:18.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:18.188 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:17 smithi097 bash[24958]: audit 2023-11-17T07:20:16.232948+0000 mgr.smithi097.swvonl (mgr.14180) 1403 : audit [DBG] from='client.15124 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:18.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:17 smithi193 bash[28559]: audit 2023-11-17T07:20:16.232948+0000 mgr.smithi097.swvonl (mgr.14180) 1403 : audit [DBG] from='client.15124 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:19.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:18 smithi097 bash[24958]: cluster 2023-11-17T07:20:17.147125+0000 mgr.smithi097.swvonl (mgr.14180) 1404 : cluster [DBG] pgmap v1010: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:19.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:18 smithi193 bash[28559]: cluster 2023-11-17T07:20:17.147125+0000 mgr.smithi097.swvonl (mgr.14180) 1404 : cluster [DBG] pgmap v1010: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:20.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:19 smithi097 bash[24958]: audit 2023-11-17T07:20:19.499557+0000 mon.smithi097 (mon.0) 1034 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.nythfr"}]: dispatch 2023-11-17T07:20:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:19 smithi193 bash[28559]: audit 2023-11-17T07:20:19.499557+0000 mon.smithi097 (mon.0) 1034 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.nythfr"}]: dispatch 2023-11-17T07:20:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cluster 2023-11-17T07:20:19.148187+0000 mgr.smithi097.swvonl (mgr.14180) 1405 : cluster [DBG] pgmap v1011: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cephadm 2023-11-17T07:20:19.498656+0000 mgr.smithi097.swvonl (mgr.14180) 1406 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:20:21.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cephadm 2023-11-17T07:20:19.499098+0000 mgr.smithi097.swvonl (mgr.14180) 1407 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cephadm 2023-11-17T07:20:19.500528+0000 mgr.smithi097.swvonl (mgr.14180) 1408 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cephadm 2023-11-17T07:20:19.502329+0000 mgr.smithi097.swvonl (mgr.14180) 1409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cephadm 2023-11-17T07:20:19.503851+0000 mgr.smithi097.swvonl (mgr.14180) 1410 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cluster 2023-11-17T07:20:19.504520+0000 mgr.smithi097.swvonl (mgr.14180) 1411 : cluster [DBG] pgmap v1012: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:20:21.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:20 smithi097 bash[24958]: cluster 2023-11-17T07:20:19.897972+0000 mon.smithi097 (mon.0) 1035 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:20:21.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cluster 2023-11-17T07:20:19.148187+0000 mgr.smithi097.swvonl (mgr.14180) 1405 : cluster [DBG] pgmap v1011: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cephadm 2023-11-17T07:20:19.498656+0000 mgr.smithi097.swvonl (mgr.14180) 1406 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cephadm 2023-11-17T07:20:19.499098+0000 mgr.smithi097.swvonl (mgr.14180) 1407 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cephadm 2023-11-17T07:20:19.500528+0000 mgr.smithi097.swvonl (mgr.14180) 1408 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nythfr ... 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cephadm 2023-11-17T07:20:19.502329+0000 mgr.smithi097.swvonl (mgr.14180) 1409 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:20:21.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cephadm 2023-11-17T07:20:19.503851+0000 mgr.smithi097.swvonl (mgr.14180) 1410 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:20:21.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cluster 2023-11-17T07:20:19.504520+0000 mgr.smithi097.swvonl (mgr.14180) 1411 : cluster [DBG] pgmap v1012: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2023-11-17T07:20:21.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:20 smithi193 bash[28559]: cluster 2023-11-17T07:20:19.897972+0000 mon.smithi097 (mon.0) 1035 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:20:22.076 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:22.076 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:23.005 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:22 smithi097 bash[24958]: cluster 2023-11-17T07:20:21.505441+0000 mgr.smithi097.swvonl (mgr.14180) 1412 : cluster [DBG] pgmap v1013: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:23.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:22 smithi193 bash[28559]: cluster 2023-11-17T07:20:21.505441+0000 mgr.smithi097.swvonl (mgr.14180) 1412 : cluster [DBG] pgmap v1013: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:24.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:24.203 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:23 smithi097 bash[24958]: audit 2023-11-17T07:20:22.064826+0000 mgr.smithi097.swvonl (mgr.14180) 1413 : audit [DBG] from='client.15128 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:23 smithi193 bash[28559]: audit 2023-11-17T07:20:22.064826+0000 mgr.smithi097.swvonl (mgr.14180) 1413 : audit [DBG] from='client.15128 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:25.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:24 smithi193 bash[28559]: cluster 2023-11-17T07:20:23.506314+0000 mgr.smithi097.swvonl (mgr.14180) 1414 : cluster [DBG] pgmap v1014: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:25.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:24 smithi193 bash[28559]: audit 2023-11-17T07:20:24.390802+0000 mon.smithi097 (mon.0) 1036 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:25.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:24 smithi097 bash[24958]: cluster 2023-11-17T07:20:23.506314+0000 mgr.smithi097.swvonl (mgr.14180) 1414 : cluster [DBG] pgmap v1014: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:24 smithi097 bash[24958]: audit 2023-11-17T07:20:24.390802+0000 mon.smithi097 (mon.0) 1036 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:20:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:26 smithi097 bash[24958]: cluster 2023-11-17T07:20:25.507834+0000 mgr.smithi097.swvonl (mgr.14180) 1415 : cluster [DBG] pgmap v1015: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:27.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:26 smithi193 bash[28559]: cluster 2023-11-17T07:20:25.507834+0000 mgr.smithi097.swvonl (mgr.14180) 1415 : cluster [DBG] pgmap v1015: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:27.780 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:27.780 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:28.746 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:28 smithi097 bash[24958]: cluster 2023-11-17T07:20:27.508918+0000 mgr.smithi097.swvonl (mgr.14180) 1416 : cluster [DBG] pgmap v1016: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:28 smithi097 bash[24958]: audit 2023-11-17T07:20:27.769810+0000 mgr.smithi097.swvonl (mgr.14180) 1417 : audit [DBG] from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:29.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:28 smithi193 bash[28559]: cluster 2023-11-17T07:20:27.508918+0000 mgr.smithi097.swvonl (mgr.14180) 1416 : cluster [DBG] pgmap v1016: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:29.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:28 smithi193 bash[28559]: audit 2023-11-17T07:20:27.769810+0000 mgr.smithi097.swvonl (mgr.14180) 1417 : audit [DBG] from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:29.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:31.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:30 smithi097 bash[24958]: cluster 2023-11-17T07:20:29.510177+0000 mgr.smithi097.swvonl (mgr.14180) 1418 : cluster [DBG] pgmap v1017: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:31.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:30 smithi193 bash[28559]: cluster 2023-11-17T07:20:29.510177+0000 mgr.smithi097.swvonl (mgr.14180) 1418 : cluster [DBG] pgmap v1017: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:20:33.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:32 smithi097 bash[24958]: cluster 2023-11-17T07:20:31.510955+0000 mgr.smithi097.swvonl (mgr.14180) 1419 : cluster [DBG] pgmap v1018: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:33.201 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:33.201 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:33.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:32 smithi193 bash[28559]: cluster 2023-11-17T07:20:31.510955+0000 mgr.smithi097.swvonl (mgr.14180) 1419 : cluster [DBG] pgmap v1018: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:34.142 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:35.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:35.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:34 smithi193 bash[28559]: audit 2023-11-17T07:20:33.194964+0000 mgr.smithi097.swvonl (mgr.14180) 1420 : audit [DBG] from='client.15136 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:34 smithi193 bash[28559]: cluster 2023-11-17T07:20:33.511527+0000 mgr.smithi097.swvonl (mgr.14180) 1421 : cluster [DBG] pgmap v1019: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:35.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:34 smithi097 bash[24958]: audit 2023-11-17T07:20:33.194964+0000 mgr.smithi097.swvonl (mgr.14180) 1420 : audit [DBG] from='client.15136 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:34 smithi097 bash[24958]: cluster 2023-11-17T07:20:33.511527+0000 mgr.smithi097.swvonl (mgr.14180) 1421 : cluster [DBG] pgmap v1019: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:37.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:36 smithi097 bash[24958]: cluster 2023-11-17T07:20:35.512914+0000 mgr.smithi097.swvonl (mgr.14180) 1422 : cluster [DBG] pgmap v1020: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:36 smithi193 bash[28559]: cluster 2023-11-17T07:20:35.512914+0000 mgr.smithi097.swvonl (mgr.14180) 1422 : cluster [DBG] pgmap v1020: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:38.815 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:38.815 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:39.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:38 smithi097 bash[24958]: cluster 2023-11-17T07:20:37.513874+0000 mgr.smithi097.swvonl (mgr.14180) 1423 : cluster [DBG] pgmap v1021: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:39.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:38 smithi193 bash[28559]: cluster 2023-11-17T07:20:37.513874+0000 mgr.smithi097.swvonl (mgr.14180) 1423 : cluster [DBG] pgmap v1021: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:39.663 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:20:40.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:39 smithi097 bash[24958]: audit 2023-11-17T07:20:38.809511+0000 mgr.smithi097.swvonl (mgr.14180) 1424 : audit [DBG] from='client.15140 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:39 smithi193 bash[28559]: audit 2023-11-17T07:20:38.809511+0000 mgr.smithi097.swvonl (mgr.14180) 1424 : audit [DBG] from='client.15140 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:20:40.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:20:41.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:40 smithi097 bash[24958]: cluster 2023-11-17T07:20:39.515032+0000 mgr.smithi097.swvonl (mgr.14180) 1425 : cluster [DBG] pgmap v1022: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:41.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:40 smithi193 bash[28559]: cluster 2023-11-17T07:20:39.515032+0000 mgr.smithi097.swvonl (mgr.14180) 1425 : cluster [DBG] pgmap v1022: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:20:43.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:42 smithi097 bash[24958]: cluster 2023-11-17T07:20:41.516283+0000 mgr.smithi097.swvonl (mgr.14180) 1426 : cluster [DBG] pgmap v1023: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:43.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:42 smithi193 bash[28559]: cluster 2023-11-17T07:20:41.516283+0000 mgr.smithi097.swvonl (mgr.14180) 1426 : cluster [DBG] pgmap v1023: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:20:44.500 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:20:44.500 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:20:45.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:44 smithi097 bash[24958]: cluster 2023-11-17T07:20:43.516884+0000 mgr.smithi097.swvonl (mgr.14180) 1427 : cluster [DBG] pgmap v1024: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.833 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:44 smithi193 bash[28559]: cluster 2023-11-17T07:20:43.516884+0000 mgr.smithi097.swvonl (mgr.14180) 1427 : cluster [DBG] pgmap v1024: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:45 smithi193 bash[28559]: audit 2023-11-17T07:20:44.490137+0000 mgr.smithi097.swvonl (mgr.14180) 1428 : audit [DBG] from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:47 smithi193 bash[28559]: cluster 2023-11-17T07:20:45.518312+0000 mgr.smithi097.swvonl (mgr.14180) 1429 : cluster [DBG] pgmap v1025: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:48 smithi193 bash[28559]: cluster 2023-11-17T07:20:47.519217+0000 mgr.smithi097.swvonl (mgr.14180) 1430 : cluster [DBG] pgmap v1026: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:50 smithi193 bash[28559]: cluster 2023-11-17T07:20:49.520295+0000 mgr.smithi097.swvonl (mgr.14180) 1431 : cluster [DBG] pgmap v1027: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:52 smithi193 bash[28559]: cluster 2023-11-17T07:20:51.521595+0000 mgr.smithi097.swvonl (mgr.14180) 1432 : cluster [DBG] pgmap v1028: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:54 smithi193 bash[28559]: cluster 2023-11-17T07:20:53.522746+0000 mgr.smithi097.swvonl (mgr.14180) 1433 : cluster [DBG] pgmap v1029: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:56 smithi193 bash[28559]: cluster 2023-11-17T07:20:55.524128+0000 mgr.smithi097.swvonl (mgr.14180) 1434 : cluster [DBG] pgmap v1030: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:20:58 smithi193 bash[28559]: cluster 2023-11-17T07:20:57.525081+0000 mgr.smithi097.swvonl (mgr.14180) 1435 : cluster [DBG] pgmap v1031: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:00 smithi193 bash[28559]: cluster 2023-11-17T07:20:59.526175+0000 mgr.smithi097.swvonl (mgr.14180) 1436 : cluster [DBG] pgmap v1032: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:02 smithi193 bash[28559]: cluster 2023-11-17T07:21:01.527483+0000 mgr.smithi097.swvonl (mgr.14180) 1437 : cluster [DBG] pgmap v1033: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:05.843 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:04 smithi193 bash[28559]: cluster 2023-11-17T07:21:03.528546+0000 mgr.smithi097.swvonl (mgr.14180) 1438 : cluster [DBG] pgmap v1034: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:45 smithi097 bash[24958]: audit 2023-11-17T07:20:44.490137+0000 mgr.smithi097.swvonl (mgr.14180) 1428 : audit [DBG] from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:47 smithi097 bash[24958]: cluster 2023-11-17T07:20:45.518312+0000 mgr.smithi097.swvonl (mgr.14180) 1429 : cluster [DBG] pgmap v1025: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:48 smithi097 bash[24958]: cluster 2023-11-17T07:20:47.519217+0000 mgr.smithi097.swvonl (mgr.14180) 1430 : cluster [DBG] pgmap v1026: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:50 smithi097 bash[24958]: cluster 2023-11-17T07:20:49.520295+0000 mgr.smithi097.swvonl (mgr.14180) 1431 : cluster [DBG] pgmap v1027: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:52 smithi097 bash[24958]: cluster 2023-11-17T07:20:51.521595+0000 mgr.smithi097.swvonl (mgr.14180) 1432 : cluster [DBG] pgmap v1028: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:54 smithi097 bash[24958]: cluster 2023-11-17T07:20:53.522746+0000 mgr.smithi097.swvonl (mgr.14180) 1433 : cluster [DBG] pgmap v1029: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.844 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:56 smithi097 bash[24958]: cluster 2023-11-17T07:20:55.524128+0000 mgr.smithi097.swvonl (mgr.14180) 1434 : cluster [DBG] pgmap v1030: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.845 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:20:58 smithi097 bash[24958]: cluster 2023-11-17T07:20:57.525081+0000 mgr.smithi097.swvonl (mgr.14180) 1435 : cluster [DBG] pgmap v1031: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.845 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:00 smithi097 bash[24958]: cluster 2023-11-17T07:20:59.526175+0000 mgr.smithi097.swvonl (mgr.14180) 1436 : cluster [DBG] pgmap v1032: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:05.845 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:02 smithi097 bash[24958]: cluster 2023-11-17T07:21:01.527483+0000 mgr.smithi097.swvonl (mgr.14180) 1437 : cluster [DBG] pgmap v1033: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:05.845 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:04 smithi097 bash[24958]: cluster 2023-11-17T07:21:03.528546+0000 mgr.smithi097.swvonl (mgr.14180) 1438 : cluster [DBG] pgmap v1034: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:06.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:07.044 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:06 smithi097 bash[24958]: cluster 2023-11-17T07:21:05.529890+0000 mgr.smithi097.swvonl (mgr.14180) 1439 : cluster [DBG] pgmap v1035: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:07.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:06 smithi193 bash[28559]: cluster 2023-11-17T07:21:05.529890+0000 mgr.smithi097.swvonl (mgr.14180) 1439 : cluster [DBG] pgmap v1035: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:08.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:08 smithi097 bash[24958]: cluster 2023-11-17T07:21:07.530481+0000 mgr.smithi097.swvonl (mgr.14180) 1440 : cluster [DBG] pgmap v1036: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:09.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:08 smithi193 bash[28559]: cluster 2023-11-17T07:21:07.530481+0000 mgr.smithi097.swvonl (mgr.14180) 1440 : cluster [DBG] pgmap v1036: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:10.562 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:10.562 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:11.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:10 smithi097 bash[24958]: cluster 2023-11-17T07:21:09.531480+0000 mgr.smithi097.swvonl (mgr.14180) 1441 : cluster [DBG] pgmap v1037: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:11.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:10 smithi193 bash[28559]: cluster 2023-11-17T07:21:09.531480+0000 mgr.smithi097.swvonl (mgr.14180) 1441 : cluster [DBG] pgmap v1037: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:11.492 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:11.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:11 smithi097 bash[24958]: audit 2023-11-17T07:21:10.551444+0000 mgr.smithi097.swvonl (mgr.14180) 1442 : audit [DBG] from='client.15148 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:12.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:11 smithi193 bash[28559]: audit 2023-11-17T07:21:10.551444+0000 mgr.smithi097.swvonl (mgr.14180) 1442 : audit [DBG] from='client.15148 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:12.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:12.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:12 smithi097 bash[24958]: cluster 2023-11-17T07:21:11.532912+0000 mgr.smithi097.swvonl (mgr.14180) 1443 : cluster [DBG] pgmap v1038: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:13.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:12 smithi193 bash[28559]: cluster 2023-11-17T07:21:11.532912+0000 mgr.smithi097.swvonl (mgr.14180) 1443 : cluster [DBG] pgmap v1038: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:15.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:14 smithi097 bash[24958]: cluster 2023-11-17T07:21:13.534035+0000 mgr.smithi097.swvonl (mgr.14180) 1444 : cluster [DBG] pgmap v1039: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:14 smithi193 bash[28559]: cluster 2023-11-17T07:21:13.534035+0000 mgr.smithi097.swvonl (mgr.14180) 1444 : cluster [DBG] pgmap v1039: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:16.303 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:16.303 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:17.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:16 smithi097 bash[24958]: cluster 2023-11-17T07:21:15.535497+0000 mgr.smithi097.swvonl (mgr.14180) 1445 : cluster [DBG] pgmap v1040: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:16 smithi193 bash[28559]: cluster 2023-11-17T07:21:15.535497+0000 mgr.smithi097.swvonl (mgr.14180) 1445 : cluster [DBG] pgmap v1040: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:17.192 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:18.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:17 smithi097 bash[24958]: audit 2023-11-17T07:21:16.290407+0000 mgr.smithi097.swvonl (mgr.14180) 1446 : audit [DBG] from='client.15152 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:18.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:17 smithi193 bash[28559]: audit 2023-11-17T07:21:16.290407+0000 mgr.smithi097.swvonl (mgr.14180) 1446 : audit [DBG] from='client.15152 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:18.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:19.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:18 smithi097 bash[24958]: cluster 2023-11-17T07:21:17.536481+0000 mgr.smithi097.swvonl (mgr.14180) 1447 : cluster [DBG] pgmap v1041: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:19.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:18 smithi193 bash[28559]: cluster 2023-11-17T07:21:17.536481+0000 mgr.smithi097.swvonl (mgr.14180) 1447 : cluster [DBG] pgmap v1041: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:20.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:19 smithi097 bash[24958]: audit 2023-11-17T07:21:19.507388+0000 mon.smithi097 (mon.0) 1037 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:21:20.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:19 smithi193 bash[28559]: audit 2023-11-17T07:21:19.507388+0000 mon.smithi097 (mon.0) 1037 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:21:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: cluster 2023-11-17T07:21:19.537437+0000 mgr.smithi097.swvonl (mgr.14180) 1448 : cluster [DBG] pgmap v1042: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: audit 2023-11-17T07:21:19.811945+0000 mon.smithi097 (mon.0) 1038 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:21:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: audit 2023-11-17T07:21:19.812996+0000 mon.smithi097 (mon.0) 1039 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:21:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: audit 2023-11-17T07:21:19.819344+0000 mon.smithi097 (mon.0) 1040 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: cluster 2023-11-17T07:21:19.820252+0000 mgr.smithi097.swvonl (mgr.14180) 1449 : cluster [DBG] pgmap v1043: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 398 B/s rd, 0 op/s 2023-11-17T07:21:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: audit 2023-11-17T07:21:19.824679+0000 mon.smithi097 (mon.0) 1041 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:21:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: audit 2023-11-17T07:21:19.834169+0000 mon.smithi097 (mon.0) 1042 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:21.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:20 smithi097 bash[24958]: cephadm 2023-11-17T07:21:19.837758+0000 mgr.smithi097.swvonl (mgr.14180) 1450 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.dymmyl on smithi193 2023-11-17T07:21:21.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: cluster 2023-11-17T07:21:19.537437+0000 mgr.smithi097.swvonl (mgr.14180) 1448 : cluster [DBG] pgmap v1042: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: audit 2023-11-17T07:21:19.811945+0000 mon.smithi097 (mon.0) 1038 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: audit 2023-11-17T07:21:19.812996+0000 mon.smithi097 (mon.0) 1039 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: audit 2023-11-17T07:21:19.819344+0000 mon.smithi097 (mon.0) 1040 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: cluster 2023-11-17T07:21:19.820252+0000 mgr.smithi097.swvonl (mgr.14180) 1449 : cluster [DBG] pgmap v1043: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 398 B/s rd, 0 op/s 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: audit 2023-11-17T07:21:19.824679+0000 mon.smithi097 (mon.0) 1041 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: audit 2023-11-17T07:21:19.834169+0000 mon.smithi097 (mon.0) 1042 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:21.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:20 smithi193 bash[28559]: cephadm 2023-11-17T07:21:19.837758+0000 mgr.smithi097.swvonl (mgr.14180) 1450 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.dymmyl on smithi193 2023-11-17T07:21:21.858 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:21.858 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:19:10.581861Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lyjcbf on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:22.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:21 smithi097 bash[24958]: cluster 2023-11-17T07:21:20.816906+0000 mon.smithi097 (mon.0) 1043 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:21:22.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:21 smithi097 bash[24958]: cluster 2023-11-17T07:21:20.816957+0000 mon.smithi097 (mon.0) 1044 : cluster [INF] Cluster is now healthy 2023-11-17T07:21:22.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:21 smithi193 bash[28559]: cluster 2023-11-17T07:21:20.816906+0000 mon.smithi097 (mon.0) 1043 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:21:22.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:21 smithi193 bash[28559]: cluster 2023-11-17T07:21:20.816957+0000 mon.smithi097 (mon.0) 1044 : cluster [INF] Cluster is now healthy 2023-11-17T07:21:22.688 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:23.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:22 smithi097 bash[24958]: cluster 2023-11-17T07:21:21.820973+0000 mgr.smithi097.swvonl (mgr.14180) 1451 : cluster [DBG] pgmap v1044: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:23.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:22 smithi097 bash[24958]: audit 2023-11-17T07:21:21.852614+0000 mgr.smithi097.swvonl (mgr.14180) 1452 : audit [DBG] from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:23.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:22 smithi193 bash[28559]: cluster 2023-11-17T07:21:21.820973+0000 mgr.smithi097.swvonl (mgr.14180) 1451 : cluster [DBG] pgmap v1044: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:23.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:22 smithi193 bash[28559]: audit 2023-11-17T07:21:21.852614+0000 mgr.smithi097.swvonl (mgr.14180) 1452 : audit [DBG] from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:23.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:24.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:23 smithi097 bash[24958]: audit 2023-11-17T07:21:23.731217+0000 mon.smithi097 (mon.0) 1045 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.dymmyl"}]: dispatch 2023-11-17T07:21:24.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:23 smithi193 bash[28559]: audit 2023-11-17T07:21:23.731217+0000 mon.smithi097 (mon.0) 1045 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.dymmyl"}]: dispatch 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: cephadm 2023-11-17T07:21:23.730004+0000 mgr.smithi097.swvonl (mgr.14180) 1453 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:21:25.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: cephadm 2023-11-17T07:21:23.730501+0000 mgr.smithi097.swvonl (mgr.14180) 1454 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: cephadm 2023-11-17T07:21:23.732895+0000 mgr.smithi097.swvonl (mgr.14180) 1455 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: cephadm 2023-11-17T07:21:23.737899+0000 mgr.smithi097.swvonl (mgr.14180) 1456 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.bkispt on smithi097 2023-11-17T07:21:25.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:24 smithi097 bash[24958]: cluster 2023-11-17T07:21:23.821628+0000 mgr.smithi097.swvonl (mgr.14180) 1457 : cluster [DBG] pgmap v1045: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: cephadm 2023-11-17T07:21:23.730004+0000 mgr.smithi097.swvonl (mgr.14180) 1453 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: cephadm 2023-11-17T07:21:23.730501+0000 mgr.smithi097.swvonl (mgr.14180) 1454 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: cephadm 2023-11-17T07:21:23.732895+0000 mgr.smithi097.swvonl (mgr.14180) 1455 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.dymmyl ... 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:25.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:25.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:25.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:25.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: cephadm 2023-11-17T07:21:23.737899+0000 mgr.smithi097.swvonl (mgr.14180) 1456 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.bkispt on smithi097 2023-11-17T07:21:25.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:24 smithi193 bash[28559]: cluster 2023-11-17T07:21:23.821628+0000 mgr.smithi097.swvonl (mgr.14180) 1457 : cluster [DBG] pgmap v1045: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:26 smithi097 bash[24958]: cluster 2023-11-17T07:21:25.822796+0000 mgr.smithi097.swvonl (mgr.14180) 1458 : cluster [DBG] pgmap v1046: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:26 smithi097 bash[24958]: audit 2023-11-17T07:21:26.552363+0000 mon.smithi097 (mon.0) 1046 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:27.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:26 smithi193 bash[28559]: cluster 2023-11-17T07:21:25.822796+0000 mgr.smithi097.swvonl (mgr.14180) 1458 : cluster [DBG] pgmap v1046: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:27.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:26 smithi193 bash[28559]: audit 2023-11-17T07:21:26.552363+0000 mon.smithi097 (mon.0) 1046 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:27.375 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:27.376 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:15.362783Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ppovlp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ppovlp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ppovlp\nDeploy daemon haproxy.nfs.foo.smithi193.ppovlp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.500398Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nythfr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nythfr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nythfr\nDeploy daemon haproxy.nfs.foo.smithi097.nythfr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:20:19.502203Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.frzyhf on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:28.318 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:28 smithi097 bash[24958]: audit 2023-11-17T07:21:27.365055+0000 mgr.smithi097.swvonl (mgr.14180) 1459 : audit [DBG] from='client.15160 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:28 smithi097 bash[24958]: cluster 2023-11-17T07:21:27.823316+0000 mgr.smithi097.swvonl (mgr.14180) 1460 : cluster [DBG] pgmap v1047: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:28 smithi097 bash[24958]: audit 2023-11-17T07:21:28.226820+0000 mon.smithi097 (mon.0) 1047 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.bkispt"}]: dispatch 2023-11-17T07:21:29.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:28 smithi193 bash[28559]: audit 2023-11-17T07:21:27.365055+0000 mgr.smithi097.swvonl (mgr.14180) 1459 : audit [DBG] from='client.15160 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:29.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:28 smithi193 bash[28559]: cluster 2023-11-17T07:21:27.823316+0000 mgr.smithi097.swvonl (mgr.14180) 1460 : cluster [DBG] pgmap v1047: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:21:29.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:28 smithi193 bash[28559]: audit 2023-11-17T07:21:28.226820+0000 mon.smithi097 (mon.0) 1047 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.bkispt"}]: dispatch 2023-11-17T07:21:29.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cephadm 2023-11-17T07:21:28.225989+0000 mgr.smithi097.swvonl (mgr.14180) 1461 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cephadm 2023-11-17T07:21:28.226256+0000 mgr.smithi097.swvonl (mgr.14180) 1462 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cephadm 2023-11-17T07:21:28.228036+0000 mgr.smithi097.swvonl (mgr.14180) 1463 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cephadm 2023-11-17T07:21:28.230059+0000 mgr.smithi097.swvonl (mgr.14180) 1464 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cephadm 2023-11-17T07:21:28.231663+0000 mgr.smithi097.swvonl (mgr.14180) 1465 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cluster 2023-11-17T07:21:28.232787+0000 mgr.smithi097.swvonl (mgr.14180) 1466 : cluster [DBG] pgmap v1048: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: cluster 2023-11-17T07:21:28.880659+0000 mon.smithi097 (mon.0) 1048 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:21:30.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:29 smithi097 bash[24958]: audit 2023-11-17T07:21:29.437926+0000 mon.smithi097 (mon.0) 1049 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:30.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cephadm 2023-11-17T07:21:28.225989+0000 mgr.smithi097.swvonl (mgr.14180) 1461 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:21:30.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cephadm 2023-11-17T07:21:28.226256+0000 mgr.smithi097.swvonl (mgr.14180) 1462 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cephadm 2023-11-17T07:21:28.228036+0000 mgr.smithi097.swvonl (mgr.14180) 1463 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.bkispt ... 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:21:30.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cephadm 2023-11-17T07:21:28.230059+0000 mgr.smithi097.swvonl (mgr.14180) 1464 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cephadm 2023-11-17T07:21:28.231663+0000 mgr.smithi097.swvonl (mgr.14180) 1465 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cluster 2023-11-17T07:21:28.232787+0000 mgr.smithi097.swvonl (mgr.14180) 1466 : cluster [DBG] pgmap v1048: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 235 B/s rd, 0 op/s 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: cluster 2023-11-17T07:21:28.880659+0000 mon.smithi097 (mon.0) 1048 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:21:30.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:29 smithi193 bash[28559]: audit 2023-11-17T07:21:29.437926+0000 mon.smithi097 (mon.0) 1049 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:32.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:31 smithi097 bash[24958]: cluster 2023-11-17T07:21:30.234170+0000 mgr.smithi097.swvonl (mgr.14180) 1467 : cluster [DBG] pgmap v1049: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:32.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:31 smithi193 bash[28559]: cluster 2023-11-17T07:21:30.234170+0000 mgr.smithi097.swvonl (mgr.14180) 1467 : cluster [DBG] pgmap v1049: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:33.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:33.110 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:33.931 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:34.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:33 smithi097 bash[24958]: cluster 2023-11-17T07:21:32.235139+0000 mgr.smithi097.swvonl (mgr.14180) 1468 : cluster [DBG] pgmap v1050: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:34.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:33 smithi193 bash[28559]: cluster 2023-11-17T07:21:32.235139+0000 mgr.smithi097.swvonl (mgr.14180) 1468 : cluster [DBG] pgmap v1050: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:34.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:35.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:34 smithi193 bash[28559]: audit 2023-11-17T07:21:33.098522+0000 mgr.smithi097.swvonl (mgr.14180) 1469 : audit [DBG] from='client.15164 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:35.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:34 smithi097 bash[24958]: audit 2023-11-17T07:21:33.098522+0000 mgr.smithi097.swvonl (mgr.14180) 1469 : audit [DBG] from='client.15164 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:36.281 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:35 smithi097 bash[24958]: cluster 2023-11-17T07:21:34.236273+0000 mgr.smithi097.swvonl (mgr.14180) 1470 : cluster [DBG] pgmap v1051: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:36.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:35 smithi193 bash[28559]: cluster 2023-11-17T07:21:34.236273+0000 mgr.smithi097.swvonl (mgr.14180) 1470 : cluster [DBG] pgmap v1051: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:38.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:37 smithi097 bash[24958]: cluster 2023-11-17T07:21:36.237250+0000 mgr.smithi097.swvonl (mgr.14180) 1471 : cluster [DBG] pgmap v1052: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:38.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:37 smithi193 bash[28559]: cluster 2023-11-17T07:21:36.237250+0000 mgr.smithi097.swvonl (mgr.14180) 1471 : cluster [DBG] pgmap v1052: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:38.387 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:38.387 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:39.183 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:40.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:40.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:39 smithi097 bash[24958]: cluster 2023-11-17T07:21:38.238193+0000 mgr.smithi097.swvonl (mgr.14180) 1472 : cluster [DBG] pgmap v1053: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:40.196 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:39 smithi097 bash[24958]: audit 2023-11-17T07:21:38.378174+0000 mgr.smithi097.swvonl (mgr.14180) 1473 : audit [DBG] from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:39 smithi193 bash[28559]: cluster 2023-11-17T07:21:38.238193+0000 mgr.smithi097.swvonl (mgr.14180) 1472 : cluster [DBG] pgmap v1053: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 196 B/s rd, 0 op/s 2023-11-17T07:21:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:39 smithi193 bash[28559]: audit 2023-11-17T07:21:38.378174+0000 mgr.smithi097.swvonl (mgr.14180) 1473 : audit [DBG] from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:42.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:41 smithi097 bash[24958]: cluster 2023-11-17T07:21:40.239530+0000 mgr.smithi097.swvonl (mgr.14180) 1474 : cluster [DBG] pgmap v1054: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:42.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:41 smithi097 bash[24958]: audit 2023-11-17T07:21:41.551326+0000 mon.smithi097 (mon.0) 1050 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:42.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:41 smithi193 bash[28559]: cluster 2023-11-17T07:21:40.239530+0000 mgr.smithi097.swvonl (mgr.14180) 1474 : cluster [DBG] pgmap v1054: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:42.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:41 smithi193 bash[28559]: audit 2023-11-17T07:21:41.551326+0000 mon.smithi097 (mon.0) 1050 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:21:43.987 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:43.988 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:44.240 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:43 smithi097 bash[24958]: cluster 2023-11-17T07:21:42.240506+0000 mgr.smithi097.swvonl (mgr.14180) 1475 : cluster [DBG] pgmap v1055: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:44.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:43 smithi193 bash[28559]: cluster 2023-11-17T07:21:42.240506+0000 mgr.smithi097.swvonl (mgr.14180) 1475 : cluster [DBG] pgmap v1055: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:44.850 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:44 smithi193 bash[28559]: audit 2023-11-17T07:21:43.975579+0000 mgr.smithi097.swvonl (mgr.14180) 1476 : audit [DBG] from='client.15172 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:44 smithi097 bash[24958]: audit 2023-11-17T07:21:43.975579+0000 mgr.smithi097.swvonl (mgr.14180) 1476 : audit [DBG] from='client.15172 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:45.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:46.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:45 smithi097 bash[24958]: cluster 2023-11-17T07:21:44.241420+0000 mgr.smithi097.swvonl (mgr.14180) 1477 : cluster [DBG] pgmap v1056: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:45 smithi193 bash[28559]: cluster 2023-11-17T07:21:44.241420+0000 mgr.smithi097.swvonl (mgr.14180) 1477 : cluster [DBG] pgmap v1056: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:48.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:47 smithi097 bash[24958]: cluster 2023-11-17T07:21:46.242428+0000 mgr.smithi097.swvonl (mgr.14180) 1478 : cluster [DBG] pgmap v1057: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:48.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:47 smithi193 bash[28559]: cluster 2023-11-17T07:21:46.242428+0000 mgr.smithi097.swvonl (mgr.14180) 1478 : cluster [DBG] pgmap v1057: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:49.635 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:49.635 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:50.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:49 smithi097 bash[24958]: cluster 2023-11-17T07:21:48.243384+0000 mgr.smithi097.swvonl (mgr.14180) 1479 : cluster [DBG] pgmap v1058: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:50.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:49 smithi193 bash[28559]: cluster 2023-11-17T07:21:48.243384+0000 mgr.smithi097.swvonl (mgr.14180) 1479 : cluster [DBG] pgmap v1058: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:50.513 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:51.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:50 smithi097 bash[24958]: audit 2023-11-17T07:21:49.622629+0000 mgr.smithi097.swvonl (mgr.14180) 1480 : audit [DBG] from='client.15176 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:51.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:50 smithi193 bash[28559]: audit 2023-11-17T07:21:49.622629+0000 mgr.smithi097.swvonl (mgr.14180) 1480 : audit [DBG] from='client.15176 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:51.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:52.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:51 smithi097 bash[24958]: cluster 2023-11-17T07:21:50.244188+0000 mgr.smithi097.swvonl (mgr.14180) 1481 : cluster [DBG] pgmap v1059: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:51 smithi193 bash[28559]: cluster 2023-11-17T07:21:50.244188+0000 mgr.smithi097.swvonl (mgr.14180) 1481 : cluster [DBG] pgmap v1059: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:21:54.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:53 smithi097 bash[24958]: cluster 2023-11-17T07:21:52.245142+0000 mgr.smithi097.swvonl (mgr.14180) 1482 : cluster [DBG] pgmap v1060: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:53 smithi193 bash[28559]: cluster 2023-11-17T07:21:52.245142+0000 mgr.smithi097.swvonl (mgr.14180) 1482 : cluster [DBG] pgmap v1060: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:21:55.228 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:21:55.976 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:21:56.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:55 smithi097 bash[24958]: cluster 2023-11-17T07:21:54.246302+0000 mgr.smithi097.swvonl (mgr.14180) 1483 : cluster [DBG] pgmap v1061: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:56.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:55 smithi193 bash[28559]: cluster 2023-11-17T07:21:54.246302+0000 mgr.smithi097.swvonl (mgr.14180) 1483 : cluster [DBG] pgmap v1061: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:56.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:21:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:56 smithi097 bash[24958]: audit 2023-11-17T07:21:55.216160+0000 mgr.smithi097.swvonl (mgr.14180) 1484 : audit [DBG] from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:57.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:56 smithi193 bash[28559]: audit 2023-11-17T07:21:55.216160+0000 mgr.smithi097.swvonl (mgr.14180) 1484 : audit [DBG] from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:21:58.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:21:58 smithi097 bash[24958]: cluster 2023-11-17T07:21:56.247708+0000 mgr.smithi097.swvonl (mgr.14180) 1485 : cluster [DBG] pgmap v1062: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:21:58.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:21:58 smithi193 bash[28559]: cluster 2023-11-17T07:21:56.247708+0000 mgr.smithi097.swvonl (mgr.14180) 1485 : cluster [DBG] pgmap v1062: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:00.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:00 smithi097 bash[24958]: cluster 2023-11-17T07:21:58.248622+0000 mgr.smithi097.swvonl (mgr.14180) 1486 : cluster [DBG] pgmap v1063: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:00.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:00 smithi193 bash[28559]: cluster 2023-11-17T07:21:58.248622+0000 mgr.smithi097.swvonl (mgr.14180) 1486 : cluster [DBG] pgmap v1063: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:00.428 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:00.429 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:01.286 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:02.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:02.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:02 smithi097 bash[24958]: cluster 2023-11-17T07:22:00.249993+0000 mgr.smithi097.swvonl (mgr.14180) 1487 : cluster [DBG] pgmap v1064: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:02 smithi097 bash[24958]: audit 2023-11-17T07:22:00.417564+0000 mgr.smithi097.swvonl (mgr.14180) 1488 : audit [DBG] from='client.15184 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:02.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:02 smithi193 bash[28559]: cluster 2023-11-17T07:22:00.249993+0000 mgr.smithi097.swvonl (mgr.14180) 1487 : cluster [DBG] pgmap v1064: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:02.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:02 smithi193 bash[28559]: audit 2023-11-17T07:22:00.417564+0000 mgr.smithi097.swvonl (mgr.14180) 1488 : audit [DBG] from='client.15184 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:04.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:04 smithi097 bash[24958]: cluster 2023-11-17T07:22:02.250971+0000 mgr.smithi097.swvonl (mgr.14180) 1489 : cluster [DBG] pgmap v1065: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:04.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:04 smithi193 bash[28559]: cluster 2023-11-17T07:22:02.250971+0000 mgr.smithi097.swvonl (mgr.14180) 1489 : cluster [DBG] pgmap v1065: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:05.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:05 smithi097 bash[24958]: cluster 2023-11-17T07:22:04.251799+0000 mgr.smithi097.swvonl (mgr.14180) 1490 : cluster [DBG] pgmap v1066: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:05.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:05 smithi193 bash[28559]: cluster 2023-11-17T07:22:04.251799+0000 mgr.smithi097.swvonl (mgr.14180) 1490 : cluster [DBG] pgmap v1066: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:05.654 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:05.654 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:06.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:06 smithi097 bash[24958]: audit 2023-11-17T07:22:05.644404+0000 mgr.smithi097.swvonl (mgr.14180) 1491 : audit [DBG] from='client.15188 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:06.534 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:06.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:06 smithi193 bash[28559]: audit 2023-11-17T07:22:05.644404+0000 mgr.smithi097.swvonl (mgr.14180) 1491 : audit [DBG] from='client.15188 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:07.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:08.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:07 smithi097 bash[24958]: cluster 2023-11-17T07:22:06.253032+0000 mgr.smithi097.swvonl (mgr.14180) 1492 : cluster [DBG] pgmap v1067: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:08.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:07 smithi193 bash[28559]: cluster 2023-11-17T07:22:06.253032+0000 mgr.smithi097.swvonl (mgr.14180) 1492 : cluster [DBG] pgmap v1067: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:10.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:09 smithi097 bash[24958]: cluster 2023-11-17T07:22:08.254052+0000 mgr.smithi097.swvonl (mgr.14180) 1493 : cluster [DBG] pgmap v1068: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:10.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:09 smithi193 bash[28559]: cluster 2023-11-17T07:22:08.254052+0000 mgr.smithi097.swvonl (mgr.14180) 1493 : cluster [DBG] pgmap v1068: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:10.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:10.966 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:11.875 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:12.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:11 smithi097 bash[24958]: cluster 2023-11-17T07:22:10.255434+0000 mgr.smithi097.swvonl (mgr.14180) 1494 : cluster [DBG] pgmap v1069: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:12.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:11 smithi097 bash[24958]: audit 2023-11-17T07:22:10.956474+0000 mgr.smithi097.swvonl (mgr.14180) 1495 : audit [DBG] from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:12.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:11 smithi193 bash[28559]: cluster 2023-11-17T07:22:10.255434+0000 mgr.smithi097.swvonl (mgr.14180) 1494 : cluster [DBG] pgmap v1069: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:12.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:11 smithi193 bash[28559]: audit 2023-11-17T07:22:10.956474+0000 mgr.smithi097.swvonl (mgr.14180) 1495 : audit [DBG] from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:12.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:14.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:13 smithi097 bash[24958]: cluster 2023-11-17T07:22:12.256593+0000 mgr.smithi097.swvonl (mgr.14180) 1496 : cluster [DBG] pgmap v1070: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:14.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:13 smithi193 bash[28559]: cluster 2023-11-17T07:22:12.256593+0000 mgr.smithi097.swvonl (mgr.14180) 1496 : cluster [DBG] pgmap v1070: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:16.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:15 smithi097 bash[24958]: cluster 2023-11-17T07:22:14.257586+0000 mgr.smithi097.swvonl (mgr.14180) 1497 : cluster [DBG] pgmap v1071: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:15 smithi193 bash[28559]: cluster 2023-11-17T07:22:14.257586+0000 mgr.smithi097.swvonl (mgr.14180) 1497 : cluster [DBG] pgmap v1071: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:16.545 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:16.545 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:17.405 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:18.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:17 smithi097 bash[24958]: cluster 2023-11-17T07:22:16.258945+0000 mgr.smithi097.swvonl (mgr.14180) 1498 : cluster [DBG] pgmap v1072: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:18.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:17 smithi097 bash[24958]: audit 2023-11-17T07:22:16.535116+0000 mgr.smithi097.swvonl (mgr.14180) 1499 : audit [DBG] from='client.15196 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:18.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:17 smithi193 bash[28559]: cluster 2023-11-17T07:22:16.258945+0000 mgr.smithi097.swvonl (mgr.14180) 1498 : cluster [DBG] pgmap v1072: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:18.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:17 smithi193 bash[28559]: audit 2023-11-17T07:22:16.535116+0000 mgr.smithi097.swvonl (mgr.14180) 1499 : audit [DBG] from='client.15196 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:18.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:20.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:19 smithi097 bash[24958]: cluster 2023-11-17T07:22:18.260021+0000 mgr.smithi097.swvonl (mgr.14180) 1500 : cluster [DBG] pgmap v1073: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:20.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:19 smithi193 bash[28559]: cluster 2023-11-17T07:22:18.260021+0000 mgr.smithi097.swvonl (mgr.14180) 1500 : cluster [DBG] pgmap v1073: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:22.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:21 smithi097 bash[24958]: cluster 2023-11-17T07:22:20.260811+0000 mgr.smithi097.swvonl (mgr.14180) 1501 : cluster [DBG] pgmap v1074: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:22.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:21 smithi193 bash[28559]: cluster 2023-11-17T07:22:20.260811+0000 mgr.smithi097.swvonl (mgr.14180) 1501 : cluster [DBG] pgmap v1074: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:22.155 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:22.155 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:23.031 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:24.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:24.044 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:23 smithi097 bash[24958]: audit 2023-11-17T07:22:22.145684+0000 mgr.smithi097.swvonl (mgr.14180) 1502 : audit [DBG] from='client.15200 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:24.044 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:23 smithi097 bash[24958]: cluster 2023-11-17T07:22:22.261807+0000 mgr.smithi097.swvonl (mgr.14180) 1503 : cluster [DBG] pgmap v1075: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:24.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:23 smithi193 bash[28559]: audit 2023-11-17T07:22:22.145684+0000 mgr.smithi097.swvonl (mgr.14180) 1502 : audit [DBG] from='client.15200 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:24.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:23 smithi193 bash[28559]: cluster 2023-11-17T07:22:22.261807+0000 mgr.smithi097.swvonl (mgr.14180) 1503 : cluster [DBG] pgmap v1075: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:26.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:25 smithi097 bash[24958]: cluster 2023-11-17T07:22:24.262963+0000 mgr.smithi097.swvonl (mgr.14180) 1504 : cluster [DBG] pgmap v1076: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:26.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:25 smithi193 bash[28559]: cluster 2023-11-17T07:22:24.262963+0000 mgr.smithi097.swvonl (mgr.14180) 1504 : cluster [DBG] pgmap v1076: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:27.836 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:27.868 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:20:19.503756Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kyosli on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:28.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:27 smithi097 bash[24958]: cluster 2023-11-17T07:22:26.264297+0000 mgr.smithi097.swvonl (mgr.14180) 1505 : cluster [DBG] pgmap v1077: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:27 smithi193 bash[28559]: cluster 2023-11-17T07:22:26.264297+0000 mgr.smithi097.swvonl (mgr.14180) 1505 : cluster [DBG] pgmap v1077: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:28.721 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:27.829866+0000 mgr.smithi097.swvonl (mgr.14180) 1506 : audit [DBG] from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.236527+0000 mon.smithi097 (mon.0) 1051 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.534490+0000 mon.smithi097 (mon.0) 1052 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.536083+0000 mon.smithi097 (mon.0) 1053 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.544851+0000 mon.smithi097 (mon.0) 1054 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.553664+0000 mon.smithi097 (mon.0) 1055 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:22:29.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:28 smithi097 bash[24958]: audit 2023-11-17T07:22:28.569379+0000 mon.smithi097 (mon.0) 1056 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:29.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:27.829866+0000 mgr.smithi097.swvonl (mgr.14180) 1506 : audit [DBG] from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:29.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.236527+0000 mon.smithi097 (mon.0) 1051 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:22:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.534490+0000 mon.smithi097 (mon.0) 1052 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:22:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.536083+0000 mon.smithi097 (mon.0) 1053 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:22:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.544851+0000 mon.smithi097 (mon.0) 1054 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.553664+0000 mon.smithi097 (mon.0) 1055 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:22:29.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:28 smithi193 bash[28559]: audit 2023-11-17T07:22:28.569379+0000 mon.smithi097 (mon.0) 1056 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:29.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:30.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:29 smithi097 bash[24958]: cluster 2023-11-17T07:22:28.265420+0000 mgr.smithi097.swvonl (mgr.14180) 1507 : cluster [DBG] pgmap v1078: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:30.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:29 smithi097 bash[24958]: cluster 2023-11-17T07:22:28.545510+0000 mgr.smithi097.swvonl (mgr.14180) 1508 : cluster [DBG] pgmap v1079: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:30.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:29 smithi097 bash[24958]: cephadm 2023-11-17T07:22:28.573498+0000 mgr.smithi097.swvonl (mgr.14180) 1509 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.azktwe on smithi193 2023-11-17T07:22:30.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:29 smithi097 bash[24958]: cluster 2023-11-17T07:22:28.756726+0000 mon.smithi097 (mon.0) 1057 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:22:30.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:29 smithi097 bash[24958]: cluster 2023-11-17T07:22:28.756788+0000 mon.smithi097 (mon.0) 1058 : cluster [INF] Cluster is now healthy 2023-11-17T07:22:30.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:29 smithi193 bash[28559]: cluster 2023-11-17T07:22:28.265420+0000 mgr.smithi097.swvonl (mgr.14180) 1507 : cluster [DBG] pgmap v1078: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:30.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:29 smithi193 bash[28559]: cluster 2023-11-17T07:22:28.545510+0000 mgr.smithi097.swvonl (mgr.14180) 1508 : cluster [DBG] pgmap v1079: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:30.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:29 smithi193 bash[28559]: cephadm 2023-11-17T07:22:28.573498+0000 mgr.smithi097.swvonl (mgr.14180) 1509 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.azktwe on smithi193 2023-11-17T07:22:30.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:29 smithi193 bash[28559]: cluster 2023-11-17T07:22:28.756726+0000 mon.smithi097 (mon.0) 1057 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:22:30.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:29 smithi193 bash[28559]: cluster 2023-11-17T07:22:28.756788+0000 mon.smithi097 (mon.0) 1058 : cluster [INF] Cluster is now healthy 2023-11-17T07:22:32.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:31 smithi097 bash[24958]: cluster 2023-11-17T07:22:30.546701+0000 mgr.smithi097.swvonl (mgr.14180) 1510 : cluster [DBG] pgmap v1080: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:32.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:31 smithi193 bash[28559]: cluster 2023-11-17T07:22:30.546701+0000 mgr.smithi097.swvonl (mgr.14180) 1510 : cluster [DBG] pgmap v1080: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:33.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:32 smithi097 bash[24958]: audit 2023-11-17T07:22:32.477488+0000 mon.smithi097 (mon.0) 1059 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.azktwe"}]: dispatch 2023-11-17T07:22:33.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:32 smithi193 bash[28559]: audit 2023-11-17T07:22:32.477488+0000 mon.smithi097 (mon.0) 1059 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.azktwe"}]: dispatch 2023-11-17T07:22:33.363 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:33.363 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:23.732596Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.dymmyl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-dymmyl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.dymmyl\nDeploy daemon haproxy.nfs.foo.smithi193.dymmyl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.227903Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.bkispt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-bkispt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.bkispt\nDeploy daemon haproxy.nfs.foo.smithi097.bkispt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:21:28.229964Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fgfrye on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: cephadm 2023-11-17T07:22:32.476010+0000 mgr.smithi097.swvonl (mgr.14180) 1511 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:22:34.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: cephadm 2023-11-17T07:22:32.476572+0000 mgr.smithi097.swvonl (mgr.14180) 1512 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: cephadm 2023-11-17T07:22:32.479231+0000 mgr.smithi097.swvonl (mgr.14180) 1513 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: cephadm 2023-11-17T07:22:32.484084+0000 mgr.smithi097.swvonl (mgr.14180) 1514 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.fanmii on smithi097 2023-11-17T07:22:34.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:33 smithi097 bash[24958]: cluster 2023-11-17T07:22:32.547458+0000 mgr.smithi097.swvonl (mgr.14180) 1515 : cluster [DBG] pgmap v1081: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:34.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: cephadm 2023-11-17T07:22:32.476010+0000 mgr.smithi097.swvonl (mgr.14180) 1511 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:22:34.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: cephadm 2023-11-17T07:22:32.476572+0000 mgr.smithi097.swvonl (mgr.14180) 1512 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: cephadm 2023-11-17T07:22:32.479231+0000 mgr.smithi097.swvonl (mgr.14180) 1513 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.azktwe ... 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: cephadm 2023-11-17T07:22:32.484084+0000 mgr.smithi097.swvonl (mgr.14180) 1514 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.fanmii on smithi097 2023-11-17T07:22:34.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:33 smithi193 bash[28559]: cluster 2023-11-17T07:22:32.547458+0000 mgr.smithi097.swvonl (mgr.14180) 1515 : cluster [DBG] pgmap v1081: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:34.274 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:35.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:34 smithi097 bash[24958]: audit 2023-11-17T07:22:33.352821+0000 mgr.smithi097.swvonl (mgr.14180) 1516 : audit [DBG] from='client.15208 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:35.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:34 smithi193 bash[28559]: audit 2023-11-17T07:22:33.352821+0000 mgr.smithi097.swvonl (mgr.14180) 1516 : audit [DBG] from='client.15208 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:35.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:36.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:35 smithi193 bash[28559]: cluster 2023-11-17T07:22:34.548351+0000 mgr.smithi097.swvonl (mgr.14180) 1517 : cluster [DBG] pgmap v1082: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:36.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:35 smithi097 bash[24958]: cluster 2023-11-17T07:22:34.548351+0000 mgr.smithi097.swvonl (mgr.14180) 1517 : cluster [DBG] pgmap v1082: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:22:37.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:36 smithi097 bash[24958]: audit 2023-11-17T07:22:36.447348+0000 mon.smithi097 (mon.0) 1060 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.fanmii"}]: dispatch 2023-11-17T07:22:37.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:36 smithi193 bash[28559]: audit 2023-11-17T07:22:36.447348+0000 mon.smithi097 (mon.0) 1060 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.fanmii"}]: dispatch 2023-11-17T07:22:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cephadm 2023-11-17T07:22:36.446173+0000 mgr.smithi097.swvonl (mgr.14180) 1518 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cephadm 2023-11-17T07:22:36.446684+0000 mgr.smithi097.swvonl (mgr.14180) 1519 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cephadm 2023-11-17T07:22:36.448514+0000 mgr.smithi097.swvonl (mgr.14180) 1520 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cephadm 2023-11-17T07:22:36.452735+0000 mgr.smithi097.swvonl (mgr.14180) 1521 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cephadm 2023-11-17T07:22:36.455332+0000 mgr.smithi097.swvonl (mgr.14180) 1522 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cluster 2023-11-17T07:22:36.456389+0000 mgr.smithi097.swvonl (mgr.14180) 1523 : cluster [DBG] pgmap v1083: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:22:38.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:37 smithi097 bash[24958]: cluster 2023-11-17T07:22:36.800038+0000 mon.smithi097 (mon.0) 1061 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:22:38.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cephadm 2023-11-17T07:22:36.446173+0000 mgr.smithi097.swvonl (mgr.14180) 1518 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.312 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cephadm 2023-11-17T07:22:36.446684+0000 mgr.smithi097.swvonl (mgr.14180) 1519 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cephadm 2023-11-17T07:22:36.448514+0000 mgr.smithi097.swvonl (mgr.14180) 1520 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.313 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fanmii ... 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:22:38.314 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cephadm 2023-11-17T07:22:36.452735+0000 mgr.smithi097.swvonl (mgr.14180) 1521 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cephadm 2023-11-17T07:22:36.455332+0000 mgr.smithi097.swvonl (mgr.14180) 1522 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cluster 2023-11-17T07:22:36.456389+0000 mgr.smithi097.swvonl (mgr.14180) 1523 : cluster [DBG] pgmap v1083: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:22:38.315 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:37 smithi193 bash[28559]: cluster 2023-11-17T07:22:36.800038+0000 mon.smithi097 (mon.0) 1061 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:22:39.108 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:39.109 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:39.946 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:40.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:39 smithi097 bash[24958]: cluster 2023-11-17T07:22:38.457019+0000 mgr.smithi097.swvonl (mgr.14180) 1524 : cluster [DBG] pgmap v1084: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:22:40.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:39 smithi097 bash[24958]: audit 2023-11-17T07:22:39.489560+0000 mon.smithi097 (mon.0) 1062 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:39 smithi193 bash[28559]: cluster 2023-11-17T07:22:38.457019+0000 mgr.smithi097.swvonl (mgr.14180) 1524 : cluster [DBG] pgmap v1084: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 200 B/s rd, 0 op/s 2023-11-17T07:22:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:39 smithi193 bash[28559]: audit 2023-11-17T07:22:39.489560+0000 mon.smithi097 (mon.0) 1062 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:22:40.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:41.152 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:40 smithi097 bash[24958]: audit 2023-11-17T07:22:39.097449+0000 mgr.smithi097.swvonl (mgr.14180) 1525 : audit [DBG] from='client.15212 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:41.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:40 smithi193 bash[28559]: audit 2023-11-17T07:22:39.097449+0000 mgr.smithi097.swvonl (mgr.14180) 1525 : audit [DBG] from='client.15212 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:42.220 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:41 smithi097 bash[24958]: cluster 2023-11-17T07:22:40.458445+0000 mgr.smithi097.swvonl (mgr.14180) 1526 : cluster [DBG] pgmap v1085: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:22:42.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:41 smithi193 bash[28559]: cluster 2023-11-17T07:22:40.458445+0000 mgr.smithi097.swvonl (mgr.14180) 1526 : cluster [DBG] pgmap v1085: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:22:44.261 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:43 smithi097 bash[24958]: cluster 2023-11-17T07:22:42.459302+0000 mgr.smithi097.swvonl (mgr.14180) 1527 : cluster [DBG] pgmap v1086: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:44.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:43 smithi193 bash[28559]: cluster 2023-11-17T07:22:42.459302+0000 mgr.smithi097.swvonl (mgr.14180) 1527 : cluster [DBG] pgmap v1086: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:44.703 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:45.578 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:46.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:45 smithi097 bash[24958]: cluster 2023-11-17T07:22:44.460252+0000 mgr.smithi097.swvonl (mgr.14180) 1528 : cluster [DBG] pgmap v1087: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:46.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:45 smithi097 bash[24958]: audit 2023-11-17T07:22:44.690272+0000 mgr.smithi097.swvonl (mgr.14180) 1529 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:45 smithi193 bash[28559]: cluster 2023-11-17T07:22:44.460252+0000 mgr.smithi097.swvonl (mgr.14180) 1528 : cluster [DBG] pgmap v1087: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:45 smithi193 bash[28559]: audit 2023-11-17T07:22:44.690272+0000 mgr.smithi097.swvonl (mgr.14180) 1529 : audit [DBG] from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:46.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:48.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:47 smithi097 bash[24958]: cluster 2023-11-17T07:22:46.461546+0000 mgr.smithi097.swvonl (mgr.14180) 1530 : cluster [DBG] pgmap v1088: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:48.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:47 smithi193 bash[28559]: cluster 2023-11-17T07:22:46.461546+0000 mgr.smithi097.swvonl (mgr.14180) 1530 : cluster [DBG] pgmap v1088: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:22:50.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:49 smithi097 bash[24958]: cluster 2023-11-17T07:22:48.462601+0000 mgr.smithi097.swvonl (mgr.14180) 1531 : cluster [DBG] pgmap v1089: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:50.181 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:50.181 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:50.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:49 smithi193 bash[28559]: cluster 2023-11-17T07:22:48.462601+0000 mgr.smithi097.swvonl (mgr.14180) 1531 : cluster [DBG] pgmap v1089: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:51.051 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:52.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:52.259 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:51 smithi097 bash[24958]: audit 2023-11-17T07:22:50.171420+0000 mgr.smithi097.swvonl (mgr.14180) 1532 : audit [DBG] from='client.15220 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:52.259 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:51 smithi097 bash[24958]: cluster 2023-11-17T07:22:50.463990+0000 mgr.smithi097.swvonl (mgr.14180) 1533 : cluster [DBG] pgmap v1090: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:52.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:51 smithi193 bash[28559]: audit 2023-11-17T07:22:50.171420+0000 mgr.smithi097.swvonl (mgr.14180) 1532 : audit [DBG] from='client.15220 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:52.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:51 smithi193 bash[28559]: cluster 2023-11-17T07:22:50.463990+0000 mgr.smithi097.swvonl (mgr.14180) 1533 : cluster [DBG] pgmap v1090: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:22:54.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:53 smithi097 bash[24958]: cluster 2023-11-17T07:22:52.464969+0000 mgr.smithi097.swvonl (mgr.14180) 1534 : cluster [DBG] pgmap v1091: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:54.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:53 smithi193 bash[28559]: cluster 2023-11-17T07:22:52.464969+0000 mgr.smithi097.swvonl (mgr.14180) 1534 : cluster [DBG] pgmap v1091: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:55.667 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:22:55.667 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:22:56.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:55 smithi097 bash[24958]: cluster 2023-11-17T07:22:54.466074+0000 mgr.smithi097.swvonl (mgr.14180) 1535 : cluster [DBG] pgmap v1092: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:56.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:55 smithi193 bash[28559]: cluster 2023-11-17T07:22:54.466074+0000 mgr.smithi097.swvonl (mgr.14180) 1535 : cluster [DBG] pgmap v1092: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:56.563 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:22:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:56 smithi097 bash[24958]: audit 2023-11-17T07:22:55.657465+0000 mgr.smithi097.swvonl (mgr.14180) 1536 : audit [DBG] from='client.15224 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:56 smithi193 bash[28559]: audit 2023-11-17T07:22:55.657465+0000 mgr.smithi097.swvonl (mgr.14180) 1536 : audit [DBG] from='client.15224 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:22:57.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:22:58.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:57 smithi097 bash[24958]: cluster 2023-11-17T07:22:56.467441+0000 mgr.smithi097.swvonl (mgr.14180) 1537 : cluster [DBG] pgmap v1093: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:22:58.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:57 smithi193 bash[28559]: cluster 2023-11-17T07:22:56.467441+0000 mgr.smithi097.swvonl (mgr.14180) 1537 : cluster [DBG] pgmap v1093: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:00.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:22:59 smithi097 bash[24958]: cluster 2023-11-17T07:22:58.468475+0000 mgr.smithi097.swvonl (mgr.14180) 1538 : cluster [DBG] pgmap v1094: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:00.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:22:59 smithi193 bash[28559]: cluster 2023-11-17T07:22:58.468475+0000 mgr.smithi097.swvonl (mgr.14180) 1538 : cluster [DBG] pgmap v1094: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:01.272 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:02.142 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:01 smithi097 bash[24958]: cluster 2023-11-17T07:23:00.469459+0000 mgr.smithi097.swvonl (mgr.14180) 1539 : cluster [DBG] pgmap v1095: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:01 smithi193 bash[28559]: cluster 2023-11-17T07:23:00.469459+0000 mgr.smithi097.swvonl (mgr.14180) 1539 : cluster [DBG] pgmap v1095: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:03.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:23:03.155 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:02 smithi097 bash[24958]: audit 2023-11-17T07:23:01.262200+0000 mgr.smithi097.swvonl (mgr.14180) 1540 : audit [DBG] from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:02 smithi193 bash[28559]: audit 2023-11-17T07:23:01.262200+0000 mgr.smithi097.swvonl (mgr.14180) 1540 : audit [DBG] from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:04.297 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:03 smithi097 bash[24958]: cluster 2023-11-17T07:23:02.470421+0000 mgr.smithi097.swvonl (mgr.14180) 1541 : cluster [DBG] pgmap v1096: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:04.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:03 smithi193 bash[28559]: cluster 2023-11-17T07:23:02.470421+0000 mgr.smithi097.swvonl (mgr.14180) 1541 : cluster [DBG] pgmap v1096: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:06.285 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:05 smithi097 bash[24958]: cluster 2023-11-17T07:23:04.471485+0000 mgr.smithi097.swvonl (mgr.14180) 1542 : cluster [DBG] pgmap v1097: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:06.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:05 smithi193 bash[28559]: cluster 2023-11-17T07:23:04.471485+0000 mgr.smithi097.swvonl (mgr.14180) 1542 : cluster [DBG] pgmap v1097: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:06.692 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:06.692 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:07.592 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:08.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:07 smithi097 bash[24958]: cluster 2023-11-17T07:23:06.472981+0000 mgr.smithi097.swvonl (mgr.14180) 1543 : cluster [DBG] pgmap v1098: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:08.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:07 smithi097 bash[24958]: audit 2023-11-17T07:23:06.681412+0000 mgr.smithi097.swvonl (mgr.14180) 1544 : audit [DBG] from='client.15232 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:08.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:07 smithi193 bash[28559]: cluster 2023-11-17T07:23:06.472981+0000 mgr.smithi097.swvonl (mgr.14180) 1543 : cluster [DBG] pgmap v1098: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:08.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:07 smithi193 bash[28559]: audit 2023-11-17T07:23:06.681412+0000 mgr.smithi097.swvonl (mgr.14180) 1544 : audit [DBG] from='client.15232 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:08.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:23:10.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:09 smithi097 bash[24958]: cluster 2023-11-17T07:23:08.474327+0000 mgr.smithi097.swvonl (mgr.14180) 1545 : cluster [DBG] pgmap v1099: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:10.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:09 smithi193 bash[28559]: cluster 2023-11-17T07:23:08.474327+0000 mgr.smithi097.swvonl (mgr.14180) 1545 : cluster [DBG] pgmap v1099: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:12.274 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:11 smithi097 bash[24958]: cluster 2023-11-17T07:23:10.475103+0000 mgr.smithi097.swvonl (mgr.14180) 1546 : cluster [DBG] pgmap v1100: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:12.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:11 smithi193 bash[28559]: cluster 2023-11-17T07:23:10.475103+0000 mgr.smithi097.swvonl (mgr.14180) 1546 : cluster [DBG] pgmap v1100: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:13.179 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:14.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:23:14.192 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:13 smithi097 bash[24958]: audit 2023-11-17T07:23:12.230865+0000 mgr.smithi097.swvonl (mgr.14180) 1547 : audit [DBG] from='client.15236 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:14.192 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:13 smithi097 bash[24958]: cluster 2023-11-17T07:23:12.475977+0000 mgr.smithi097.swvonl (mgr.14180) 1548 : cluster [DBG] pgmap v1101: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:14.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:13 smithi193 bash[28559]: audit 2023-11-17T07:23:12.230865+0000 mgr.smithi097.swvonl (mgr.14180) 1547 : audit [DBG] from='client.15236 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:14.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:13 smithi193 bash[28559]: cluster 2023-11-17T07:23:12.475977+0000 mgr.smithi097.swvonl (mgr.14180) 1548 : cluster [DBG] pgmap v1101: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:16.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:15 smithi097 bash[24958]: cluster 2023-11-17T07:23:14.476638+0000 mgr.smithi097.swvonl (mgr.14180) 1549 : cluster [DBG] pgmap v1102: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.201 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:48.201 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:21:28.231562Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.isiqfj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:32.478995Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.azktwe on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-azktwe\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.azktwe\nDeploy daemon haproxy.nfs.foo.smithi193.azktwe ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.448298Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fanmii on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fanmii\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fanmii\nDeploy daemon haproxy.nfs.foo.smithi097.fanmii ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:22:36.452530Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.wqdvpz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:48.202 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:17 smithi097 bash[24958]: cluster 2023-11-17T07:23:16.477987+0000 mgr.smithi097.swvonl (mgr.14180) 1550 : cluster [DBG] pgmap v1103: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.204 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:15 smithi193 bash[28559]: cluster 2023-11-17T07:23:14.476638+0000 mgr.smithi097.swvonl (mgr.14180) 1549 : cluster [DBG] pgmap v1102: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:17 smithi193 bash[28559]: cluster 2023-11-17T07:23:16.477987+0000 mgr.smithi097.swvonl (mgr.14180) 1550 : cluster [DBG] pgmap v1103: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:18 smithi193 bash[28559]: audit 2023-11-17T07:23:18.021064+0000 mgr.smithi097.swvonl (mgr.14180) 1551 : audit [DBG] from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:19 smithi193 bash[28559]: cluster 2023-11-17T07:23:18.478891+0000 mgr.smithi097.swvonl (mgr.14180) 1552 : cluster [DBG] pgmap v1104: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:21 smithi193 bash[28559]: cluster 2023-11-17T07:23:20.480276+0000 mgr.smithi097.swvonl (mgr.14180) 1553 : cluster [DBG] pgmap v1105: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:23 smithi193 bash[28559]: cluster 2023-11-17T07:23:22.481370+0000 mgr.smithi097.swvonl (mgr.14180) 1554 : cluster [DBG] pgmap v1106: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:25 smithi193 bash[28559]: cluster 2023-11-17T07:23:24.482636+0000 mgr.smithi097.swvonl (mgr.14180) 1555 : cluster [DBG] pgmap v1107: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:27 smithi193 bash[28559]: cluster 2023-11-17T07:23:26.484145+0000 mgr.smithi097.swvonl (mgr.14180) 1556 : cluster [DBG] pgmap v1108: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:29 smithi193 bash[28559]: cluster 2023-11-17T07:23:28.485258+0000 mgr.smithi097.swvonl (mgr.14180) 1557 : cluster [DBG] pgmap v1109: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.207 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:31 smithi193 bash[28559]: cluster 2023-11-17T07:23:30.486718+0000 mgr.smithi097.swvonl (mgr.14180) 1558 : cluster [DBG] pgmap v1110: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:34 smithi193 bash[28559]: cluster 2023-11-17T07:23:32.487846+0000 mgr.smithi097.swvonl (mgr.14180) 1559 : cluster [DBG] pgmap v1111: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:36 smithi193 bash[28559]: cluster 2023-11-17T07:23:34.489006+0000 mgr.smithi097.swvonl (mgr.14180) 1560 : cluster [DBG] pgmap v1112: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.459394+0000 mon.smithi097 (mon.0) 1063 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.767821+0000 mon.smithi097 (mon.0) 1064 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.769425+0000 mon.smithi097 (mon.0) 1065 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.779115+0000 mon.smithi097 (mon.0) 1066 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.785948+0000 mon.smithi097 (mon.0) 1067 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:37 smithi193 bash[28559]: audit 2023-11-17T07:23:36.802609+0000 mon.smithi097 (mon.0) 1068 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:38 smithi193 bash[28559]: cluster 2023-11-17T07:23:36.490202+0000 mgr.smithi097.swvonl (mgr.14180) 1561 : cluster [DBG] pgmap v1113: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:38 smithi193 bash[28559]: cluster 2023-11-17T07:23:36.779885+0000 mgr.smithi097.swvonl (mgr.14180) 1562 : cluster [DBG] pgmap v1114: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.208 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:38 smithi193 bash[28559]: cephadm 2023-11-17T07:23:36.807478+0000 mgr.smithi097.swvonl (mgr.14180) 1563 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.pteakh on smithi193 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:38 smithi193 bash[28559]: cluster 2023-11-17T07:23:37.008524+0000 mon.smithi097 (mon.0) 1069 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:38 smithi193 bash[28559]: cluster 2023-11-17T07:23:37.008576+0000 mon.smithi097 (mon.0) 1070 : cluster [INF] Cluster is now healthy 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:40 smithi193 bash[28559]: cluster 2023-11-17T07:23:38.780869+0000 mgr.smithi097.swvonl (mgr.14180) 1564 : cluster [DBG] pgmap v1115: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:41 smithi193 bash[28559]: audit 2023-11-17T07:23:40.548569+0000 mon.smithi097 (mon.0) 1071 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.pteakh"}]: dispatch 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: cephadm 2023-11-17T07:23:40.547570+0000 mgr.smithi097.swvonl (mgr.14180) 1565 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.209 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:23:48.233 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.234 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: cephadm 2023-11-17T07:23:40.547993+0000 mgr.smithi097.swvonl (mgr.14180) 1566 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: cephadm 2023-11-17T07:23:40.550078+0000 mgr.smithi097.swvonl (mgr.14180) 1567 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.235 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: cephadm 2023-11-17T07:23:40.554379+0000 mgr.smithi097.swvonl (mgr.14180) 1568 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.uuwcbq on smithi097 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: cluster 2023-11-17T07:23:40.781795+0000 mgr.smithi097.swvonl (mgr.14180) 1569 : cluster [DBG] pgmap v1116: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:42 smithi193 bash[28559]: audit 2023-11-17T07:23:41.555578+0000 mon.smithi097 (mon.0) 1072 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:44 smithi193 bash[28559]: cluster 2023-11-17T07:23:42.782570+0000 mgr.smithi097.swvonl (mgr.14180) 1570 : cluster [DBG] pgmap v1117: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:45 smithi193 bash[28559]: audit 2023-11-17T07:23:44.331455+0000 mon.smithi097 (mon.0) 1073 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.uuwcbq"}]: dispatch 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:45 smithi193 bash[28559]: audit 2023-11-17T07:23:44.535026+0000 mon.smithi097 (mon.0) 1074 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cephadm 2023-11-17T07:23:44.330091+0000 mgr.smithi097.swvonl (mgr.14180) 1571 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.236 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.237 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cephadm 2023-11-17T07:23:44.330675+0000 mgr.smithi097.swvonl (mgr.14180) 1572 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.238 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cephadm 2023-11-17T07:23:44.332971+0000 mgr.smithi097.swvonl (mgr.14180) 1573 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cephadm 2023-11-17T07:23:44.336479+0000 mgr.smithi097.swvonl (mgr.14180) 1574 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cephadm 2023-11-17T07:23:44.339041+0000 mgr.smithi097.swvonl (mgr.14180) 1575 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:23:48.239 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cluster 2023-11-17T07:23:44.340013+0000 mgr.smithi097.swvonl (mgr.14180) 1576 : cluster [DBG] pgmap v1118: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:23:48.240 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:46 smithi193 bash[28559]: cluster 2023-11-17T07:23:45.038794+0000 mon.smithi097 (mon.0) 1075 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:23:48.240 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:47 smithi193 bash[28559]: cluster 2023-11-17T07:23:46.341464+0000 mgr.smithi097.swvonl (mgr.14180) 1577 : cluster [DBG] pgmap v1119: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:18 smithi097 bash[24958]: audit 2023-11-17T07:23:18.021064+0000 mgr.smithi097.swvonl (mgr.14180) 1551 : audit [DBG] from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:19 smithi097 bash[24958]: cluster 2023-11-17T07:23:18.478891+0000 mgr.smithi097.swvonl (mgr.14180) 1552 : cluster [DBG] pgmap v1104: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:21 smithi097 bash[24958]: cluster 2023-11-17T07:23:20.480276+0000 mgr.smithi097.swvonl (mgr.14180) 1553 : cluster [DBG] pgmap v1105: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:23 smithi097 bash[24958]: cluster 2023-11-17T07:23:22.481370+0000 mgr.smithi097.swvonl (mgr.14180) 1554 : cluster [DBG] pgmap v1106: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:25 smithi097 bash[24958]: cluster 2023-11-17T07:23:24.482636+0000 mgr.smithi097.swvonl (mgr.14180) 1555 : cluster [DBG] pgmap v1107: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:27 smithi097 bash[24958]: cluster 2023-11-17T07:23:26.484145+0000 mgr.smithi097.swvonl (mgr.14180) 1556 : cluster [DBG] pgmap v1108: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:29 smithi097 bash[24958]: cluster 2023-11-17T07:23:28.485258+0000 mgr.smithi097.swvonl (mgr.14180) 1557 : cluster [DBG] pgmap v1109: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:31 smithi097 bash[24958]: cluster 2023-11-17T07:23:30.486718+0000 mgr.smithi097.swvonl (mgr.14180) 1558 : cluster [DBG] pgmap v1110: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:34 smithi097 bash[24958]: cluster 2023-11-17T07:23:32.487846+0000 mgr.smithi097.swvonl (mgr.14180) 1559 : cluster [DBG] pgmap v1111: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.241 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:36 smithi097 bash[24958]: cluster 2023-11-17T07:23:34.489006+0000 mgr.smithi097.swvonl (mgr.14180) 1560 : cluster [DBG] pgmap v1112: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.459394+0000 mon.smithi097 (mon.0) 1063 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.767821+0000 mon.smithi097 (mon.0) 1064 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.769425+0000 mon.smithi097 (mon.0) 1065 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.779115+0000 mon.smithi097 (mon.0) 1066 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.785948+0000 mon.smithi097 (mon.0) 1067 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:37 smithi097 bash[24958]: audit 2023-11-17T07:23:36.802609+0000 mon.smithi097 (mon.0) 1068 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:38 smithi097 bash[24958]: cluster 2023-11-17T07:23:36.490202+0000 mgr.smithi097.swvonl (mgr.14180) 1561 : cluster [DBG] pgmap v1113: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:38 smithi097 bash[24958]: cluster 2023-11-17T07:23:36.779885+0000 mgr.smithi097.swvonl (mgr.14180) 1562 : cluster [DBG] pgmap v1114: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:38 smithi097 bash[24958]: cephadm 2023-11-17T07:23:36.807478+0000 mgr.smithi097.swvonl (mgr.14180) 1563 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.pteakh on smithi193 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:38 smithi097 bash[24958]: cluster 2023-11-17T07:23:37.008524+0000 mon.smithi097 (mon.0) 1069 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:38 smithi097 bash[24958]: cluster 2023-11-17T07:23:37.008576+0000 mon.smithi097 (mon.0) 1070 : cluster [INF] Cluster is now healthy 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:40 smithi097 bash[24958]: cluster 2023-11-17T07:23:38.780869+0000 mgr.smithi097.swvonl (mgr.14180) 1564 : cluster [DBG] pgmap v1115: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.242 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:41 smithi097 bash[24958]: audit 2023-11-17T07:23:40.548569+0000 mon.smithi097 (mon.0) 1071 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.pteakh"}]: dispatch 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: cephadm 2023-11-17T07:23:40.547570+0000 mgr.smithi097.swvonl (mgr.14180) 1565 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.243 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.245 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: cephadm 2023-11-17T07:23:40.547993+0000 mgr.smithi097.swvonl (mgr.14180) 1566 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: cephadm 2023-11-17T07:23:40.550078+0000 mgr.smithi097.swvonl (mgr.14180) 1567 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh 2023-11-17T07:23:48.246 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.pteakh ... 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: cephadm 2023-11-17T07:23:40.554379+0000 mgr.smithi097.swvonl (mgr.14180) 1568 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.uuwcbq on smithi097 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: cluster 2023-11-17T07:23:40.781795+0000 mgr.smithi097.swvonl (mgr.14180) 1569 : cluster [DBG] pgmap v1116: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:42 smithi097 bash[24958]: audit 2023-11-17T07:23:41.555578+0000 mon.smithi097 (mon.0) 1072 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:44 smithi097 bash[24958]: cluster 2023-11-17T07:23:42.782570+0000 mgr.smithi097.swvonl (mgr.14180) 1570 : cluster [DBG] pgmap v1117: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:45 smithi097 bash[24958]: audit 2023-11-17T07:23:44.331455+0000 mon.smithi097 (mon.0) 1073 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.uuwcbq"}]: dispatch 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:45 smithi097 bash[24958]: audit 2023-11-17T07:23:44.535026+0000 mon.smithi097 (mon.0) 1074 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cephadm 2023-11-17T07:23:44.330091+0000 mgr.smithi097.swvonl (mgr.14180) 1571 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.247 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:23:48.248 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.249 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cephadm 2023-11-17T07:23:44.330675+0000 mgr.smithi097.swvonl (mgr.14180) 1572 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cephadm 2023-11-17T07:23:44.332971+0000 mgr.smithi097.swvonl (mgr.14180) 1573 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.uuwcbq ... 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:23:48.250 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cephadm 2023-11-17T07:23:44.336479+0000 mgr.smithi097.swvonl (mgr.14180) 1574 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:23:48.251 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cephadm 2023-11-17T07:23:44.339041+0000 mgr.smithi097.swvonl (mgr.14180) 1575 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:23:48.251 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cluster 2023-11-17T07:23:44.340013+0000 mgr.smithi097.swvonl (mgr.14180) 1576 : cluster [DBG] pgmap v1118: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:23:48.251 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:46 smithi097 bash[24958]: cluster 2023-11-17T07:23:45.038794+0000 mon.smithi097 (mon.0) 1075 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:23:48.251 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:47 smithi097 bash[24958]: cluster 2023-11-17T07:23:46.341464+0000 mgr.smithi097.swvonl (mgr.14180) 1577 : cluster [DBG] pgmap v1119: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:23:49.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:23:50.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:49 smithi097 bash[24958]: cluster 2023-11-17T07:23:48.342643+0000 mgr.smithi097.swvonl (mgr.14180) 1578 : cluster [DBG] pgmap v1120: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:50.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:49 smithi193 bash[28559]: cluster 2023-11-17T07:23:48.342643+0000 mgr.smithi097.swvonl (mgr.14180) 1578 : cluster [DBG] pgmap v1120: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:52.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:51 smithi097 bash[24958]: cluster 2023-11-17T07:23:50.344047+0000 mgr.smithi097.swvonl (mgr.14180) 1579 : cluster [DBG] pgmap v1121: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 354 B/s rd, 0 op/s 2023-11-17T07:23:52.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:51 smithi193 bash[28559]: cluster 2023-11-17T07:23:50.344047+0000 mgr.smithi097.swvonl (mgr.14180) 1579 : cluster [DBG] pgmap v1121: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 354 B/s rd, 0 op/s 2023-11-17T07:23:53.085 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:53.085 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:53.899 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:54.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:53 smithi097 bash[24958]: cluster 2023-11-17T07:23:52.344771+0000 mgr.smithi097.swvonl (mgr.14180) 1580 : cluster [DBG] pgmap v1122: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:54.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:53 smithi193 bash[28559]: cluster 2023-11-17T07:23:52.344771+0000 mgr.smithi097.swvonl (mgr.14180) 1580 : cluster [DBG] pgmap v1122: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:54.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:23:55.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:54 smithi097 bash[24958]: audit 2023-11-17T07:23:53.073319+0000 mgr.smithi097.swvonl (mgr.14180) 1581 : audit [DBG] from='client.15244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:55.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:54 smithi193 bash[28559]: audit 2023-11-17T07:23:53.073319+0000 mgr.smithi097.swvonl (mgr.14180) 1581 : audit [DBG] from='client.15244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:23:56.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:55 smithi097 bash[24958]: cluster 2023-11-17T07:23:54.345848+0000 mgr.smithi097.swvonl (mgr.14180) 1582 : cluster [DBG] pgmap v1123: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:56.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:55 smithi193 bash[28559]: cluster 2023-11-17T07:23:54.345848+0000 mgr.smithi097.swvonl (mgr.14180) 1582 : cluster [DBG] pgmap v1123: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 177 B/s rd, 0 op/s 2023-11-17T07:23:57.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:57 smithi097 bash[24958]: cluster 2023-11-17T07:23:56.347194+0000 mgr.smithi097.swvonl (mgr.14180) 1583 : cluster [DBG] pgmap v1124: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:57.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:57 smithi097 bash[24958]: audit 2023-11-17T07:23:56.554529+0000 mon.smithi097 (mon.0) 1076 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:57 smithi193 bash[28559]: cluster 2023-11-17T07:23:56.347194+0000 mgr.smithi097.swvonl (mgr.14180) 1583 : cluster [DBG] pgmap v1124: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:57 smithi193 bash[28559]: audit 2023-11-17T07:23:56.554529+0000 mon.smithi097 (mon.0) 1076 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:23:58.408 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:23:58.408 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:23:59.272 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:23:59.832 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:59 smithi097 bash[24958]: cluster 2023-11-17T07:23:58.348163+0000 mgr.smithi097.swvonl (mgr.14180) 1584 : cluster [DBG] pgmap v1125: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:23:59.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:23:59 smithi097 bash[24958]: audit 2023-11-17T07:23:58.402225+0000 mgr.smithi097.swvonl (mgr.14180) 1585 : audit [DBG] from='client.15248 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:00.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:59 smithi193 bash[28559]: cluster 2023-11-17T07:23:58.348163+0000 mgr.smithi097.swvonl (mgr.14180) 1584 : cluster [DBG] pgmap v1125: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:00.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:23:59 smithi193 bash[28559]: audit 2023-11-17T07:23:58.402225+0000 mgr.smithi097.swvonl (mgr.14180) 1585 : audit [DBG] from='client.15248 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:00.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:01.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:01 smithi097 bash[24958]: cluster 2023-11-17T07:24:00.349624+0000 mgr.smithi097.swvonl (mgr.14180) 1586 : cluster [DBG] pgmap v1126: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:02.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:01 smithi193 bash[28559]: cluster 2023-11-17T07:24:00.349624+0000 mgr.smithi097.swvonl (mgr.14180) 1586 : cluster [DBG] pgmap v1126: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:03.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:03.732 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:04.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:03 smithi097 bash[24958]: cluster 2023-11-17T07:24:02.350560+0000 mgr.smithi097.swvonl (mgr.14180) 1587 : cluster [DBG] pgmap v1127: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:04.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:03 smithi193 bash[28559]: cluster 2023-11-17T07:24:02.350560+0000 mgr.smithi097.swvonl (mgr.14180) 1587 : cluster [DBG] pgmap v1127: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:04.599 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:05.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:04 smithi097 bash[24958]: audit 2023-11-17T07:24:03.725684+0000 mgr.smithi097.swvonl (mgr.14180) 1588 : audit [DBG] from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:05.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:04 smithi193 bash[28559]: audit 2023-11-17T07:24:03.725684+0000 mgr.smithi097.swvonl (mgr.14180) 1588 : audit [DBG] from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:05.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:06.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:05 smithi097 bash[24958]: cluster 2023-11-17T07:24:04.351793+0000 mgr.smithi097.swvonl (mgr.14180) 1589 : cluster [DBG] pgmap v1128: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:06.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:05 smithi193 bash[28559]: cluster 2023-11-17T07:24:04.351793+0000 mgr.smithi097.swvonl (mgr.14180) 1589 : cluster [DBG] pgmap v1128: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:08.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:07 smithi097 bash[24958]: cluster 2023-11-17T07:24:06.353643+0000 mgr.smithi097.swvonl (mgr.14180) 1590 : cluster [DBG] pgmap v1129: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:08.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:07 smithi193 bash[28559]: cluster 2023-11-17T07:24:06.353643+0000 mgr.smithi097.swvonl (mgr.14180) 1590 : cluster [DBG] pgmap v1129: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:09.398 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:09.398 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:10.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:09 smithi097 bash[24958]: cluster 2023-11-17T07:24:08.354791+0000 mgr.smithi097.swvonl (mgr.14180) 1591 : cluster [DBG] pgmap v1130: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:10.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:09 smithi193 bash[28559]: cluster 2023-11-17T07:24:08.354791+0000 mgr.smithi097.swvonl (mgr.14180) 1591 : cluster [DBG] pgmap v1130: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:10.322 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:11.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:10 smithi097 bash[24958]: audit 2023-11-17T07:24:09.387995+0000 mgr.smithi097.swvonl (mgr.14180) 1592 : audit [DBG] from='client.15256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:11.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:10 smithi193 bash[28559]: audit 2023-11-17T07:24:09.387995+0000 mgr.smithi097.swvonl (mgr.14180) 1592 : audit [DBG] from='client.15256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:11.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:11 smithi097 bash[24958]: cluster 2023-11-17T07:24:10.356245+0000 mgr.smithi097.swvonl (mgr.14180) 1593 : cluster [DBG] pgmap v1131: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:11 smithi193 bash[28559]: cluster 2023-11-17T07:24:10.356245+0000 mgr.smithi097.swvonl (mgr.14180) 1593 : cluster [DBG] pgmap v1131: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:14.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:13 smithi097 bash[24958]: cluster 2023-11-17T07:24:12.357254+0000 mgr.smithi097.swvonl (mgr.14180) 1594 : cluster [DBG] pgmap v1132: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:14.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:13 smithi193 bash[28559]: cluster 2023-11-17T07:24:12.357254+0000 mgr.smithi097.swvonl (mgr.14180) 1594 : cluster [DBG] pgmap v1132: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:14.822 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:14.822 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:15.689 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:16.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:15 smithi097 bash[24958]: cluster 2023-11-17T07:24:14.357990+0000 mgr.smithi097.swvonl (mgr.14180) 1595 : cluster [DBG] pgmap v1133: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:16.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:15 smithi097 bash[24958]: audit 2023-11-17T07:24:14.810802+0000 mgr.smithi097.swvonl (mgr.14180) 1596 : audit [DBG] from='client.15260 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:15 smithi193 bash[28559]: cluster 2023-11-17T07:24:14.357990+0000 mgr.smithi097.swvonl (mgr.14180) 1595 : cluster [DBG] pgmap v1133: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:16.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:15 smithi193 bash[28559]: audit 2023-11-17T07:24:14.810802+0000 mgr.smithi097.swvonl (mgr.14180) 1596 : audit [DBG] from='client.15260 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:16.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:18.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:17 smithi193 bash[28559]: cluster 2023-11-17T07:24:16.359385+0000 mgr.smithi097.swvonl (mgr.14180) 1597 : cluster [DBG] pgmap v1134: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:18.126 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:17 smithi097 bash[24958]: cluster 2023-11-17T07:24:16.359385+0000 mgr.smithi097.swvonl (mgr.14180) 1597 : cluster [DBG] pgmap v1134: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:20.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:19 smithi193 bash[28559]: cluster 2023-11-17T07:24:18.360541+0000 mgr.smithi097.swvonl (mgr.14180) 1598 : cluster [DBG] pgmap v1135: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:20.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:19 smithi097 bash[24958]: cluster 2023-11-17T07:24:18.360541+0000 mgr.smithi097.swvonl (mgr.14180) 1598 : cluster [DBG] pgmap v1135: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:20.368 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:20.369 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:21.207 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:22.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:22.220 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:21 smithi097 bash[24958]: audit 2023-11-17T07:24:20.356570+0000 mgr.smithi097.swvonl (mgr.14180) 1599 : audit [DBG] from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:22.220 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:21 smithi097 bash[24958]: cluster 2023-11-17T07:24:20.362029+0000 mgr.smithi097.swvonl (mgr.14180) 1600 : cluster [DBG] pgmap v1136: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:22.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:21 smithi193 bash[28559]: audit 2023-11-17T07:24:20.356570+0000 mgr.smithi097.swvonl (mgr.14180) 1599 : audit [DBG] from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:22.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:21 smithi193 bash[28559]: cluster 2023-11-17T07:24:20.362029+0000 mgr.smithi097.swvonl (mgr.14180) 1600 : cluster [DBG] pgmap v1136: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:24.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:23 smithi097 bash[24958]: cluster 2023-11-17T07:24:22.363055+0000 mgr.smithi097.swvonl (mgr.14180) 1601 : cluster [DBG] pgmap v1137: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:23 smithi193 bash[28559]: cluster 2023-11-17T07:24:22.363055+0000 mgr.smithi097.swvonl (mgr.14180) 1601 : cluster [DBG] pgmap v1137: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:25.877 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:25.877 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:26.152 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:25 smithi097 bash[24958]: cluster 2023-11-17T07:24:24.364103+0000 mgr.smithi097.swvonl (mgr.14180) 1602 : cluster [DBG] pgmap v1138: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:26.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:25 smithi193 bash[28559]: cluster 2023-11-17T07:24:24.364103+0000 mgr.smithi097.swvonl (mgr.14180) 1602 : cluster [DBG] pgmap v1138: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:26.763 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:27.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:26 smithi097 bash[24958]: audit 2023-11-17T07:24:25.866034+0000 mgr.smithi097.swvonl (mgr.14180) 1603 : audit [DBG] from='client.15268 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:27.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:26 smithi193 bash[28559]: audit 2023-11-17T07:24:25.866034+0000 mgr.smithi097.swvonl (mgr.14180) 1603 : audit [DBG] from='client.15268 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:27.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:28.047 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:27 smithi097 bash[24958]: cluster 2023-11-17T07:24:26.365030+0000 mgr.smithi097.swvonl (mgr.14180) 1604 : cluster [DBG] pgmap v1139: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:28.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:27 smithi193 bash[28559]: cluster 2023-11-17T07:24:26.365030+0000 mgr.smithi097.swvonl (mgr.14180) 1604 : cluster [DBG] pgmap v1139: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:30.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:29 smithi097 bash[24958]: cluster 2023-11-17T07:24:28.365546+0000 mgr.smithi097.swvonl (mgr.14180) 1605 : cluster [DBG] pgmap v1140: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:30.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:29 smithi193 bash[28559]: cluster 2023-11-17T07:24:28.365546+0000 mgr.smithi097.swvonl (mgr.14180) 1605 : cluster [DBG] pgmap v1140: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:31.407 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:31.408 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:32.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:31 smithi097 bash[24958]: cluster 2023-11-17T07:24:30.366919+0000 mgr.smithi097.swvonl (mgr.14180) 1606 : cluster [DBG] pgmap v1141: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:32.251 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:32.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:31 smithi193 bash[28559]: cluster 2023-11-17T07:24:30.366919+0000 mgr.smithi097.swvonl (mgr.14180) 1606 : cluster [DBG] pgmap v1141: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:33.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:33.264 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:32 smithi097 bash[24958]: audit 2023-11-17T07:24:31.394564+0000 mgr.smithi097.swvonl (mgr.14180) 1607 : audit [DBG] from='client.15272 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:33.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:32 smithi193 bash[28559]: audit 2023-11-17T07:24:31.394564+0000 mgr.smithi097.swvonl (mgr.14180) 1607 : audit [DBG] from='client.15272 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:34.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:33 smithi097 bash[24958]: cluster 2023-11-17T07:24:32.367805+0000 mgr.smithi097.swvonl (mgr.14180) 1608 : cluster [DBG] pgmap v1142: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:33 smithi193 bash[28559]: cluster 2023-11-17T07:24:32.367805+0000 mgr.smithi097.swvonl (mgr.14180) 1608 : cluster [DBG] pgmap v1142: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:36.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:35 smithi097 bash[24958]: cluster 2023-11-17T07:24:34.368834+0000 mgr.smithi097.swvonl (mgr.14180) 1609 : cluster [DBG] pgmap v1143: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:35 smithi193 bash[28559]: cluster 2023-11-17T07:24:34.368834+0000 mgr.smithi097.swvonl (mgr.14180) 1609 : cluster [DBG] pgmap v1143: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:36.860 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:36.860 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:37.773 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:37 smithi097 bash[24958]: cluster 2023-11-17T07:24:36.370234+0000 mgr.smithi097.swvonl (mgr.14180) 1610 : cluster [DBG] pgmap v1144: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:37 smithi097 bash[24958]: audit 2023-11-17T07:24:36.848241+0000 mgr.smithi097.swvonl (mgr.14180) 1611 : audit [DBG] from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:38.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:37 smithi193 bash[28559]: cluster 2023-11-17T07:24:36.370234+0000 mgr.smithi097.swvonl (mgr.14180) 1610 : cluster [DBG] pgmap v1144: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:38.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:37 smithi193 bash[28559]: audit 2023-11-17T07:24:36.848241+0000 mgr.smithi097.swvonl (mgr.14180) 1611 : audit [DBG] from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:38.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:40.147 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:39 smithi097 bash[24958]: cluster 2023-11-17T07:24:38.371267+0000 mgr.smithi097.swvonl (mgr.14180) 1612 : cluster [DBG] pgmap v1145: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:39 smithi193 bash[28559]: cluster 2023-11-17T07:24:38.371267+0000 mgr.smithi097.swvonl (mgr.14180) 1612 : cluster [DBG] pgmap v1145: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:42.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:41 smithi097 bash[24958]: cluster 2023-11-17T07:24:40.372835+0000 mgr.smithi097.swvonl (mgr.14180) 1613 : cluster [DBG] pgmap v1146: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:42.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:41 smithi193 bash[28559]: cluster 2023-11-17T07:24:40.372835+0000 mgr.smithi097.swvonl (mgr.14180) 1613 : cluster [DBG] pgmap v1146: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:24:42.444 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:42.444 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:43.260 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:44.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:44.275 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:43 smithi097 bash[24958]: cluster 2023-11-17T07:24:42.373808+0000 mgr.smithi097.swvonl (mgr.14180) 1614 : cluster [DBG] pgmap v1147: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:44.275 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:43 smithi097 bash[24958]: audit 2023-11-17T07:24:42.434071+0000 mgr.smithi097.swvonl (mgr.14180) 1615 : audit [DBG] from='client.15280 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:44.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:43 smithi193 bash[28559]: cluster 2023-11-17T07:24:42.373808+0000 mgr.smithi097.swvonl (mgr.14180) 1614 : cluster [DBG] pgmap v1147: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:44.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:43 smithi193 bash[28559]: audit 2023-11-17T07:24:42.434071+0000 mgr.smithi097.swvonl (mgr.14180) 1615 : audit [DBG] from='client.15280 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.342922+0000 mon.smithi097 (mon.0) 1077 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.644417+0000 mon.smithi097 (mon.0) 1078 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.645859+0000 mon.smithi097 (mon.0) 1079 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.652063+0000 mon.smithi097 (mon.0) 1080 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.658941+0000 mon.smithi097 (mon.0) 1081 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:24:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:44 smithi193 bash[28559]: audit 2023-11-17T07:24:44.669105+0000 mon.smithi097 (mon.0) 1082 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.342922+0000 mon.smithi097 (mon.0) 1077 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.644417+0000 mon.smithi097 (mon.0) 1078 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.645859+0000 mon.smithi097 (mon.0) 1079 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.652063+0000 mon.smithi097 (mon.0) 1080 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.658941+0000 mon.smithi097 (mon.0) 1081 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:24:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:44 smithi097 bash[24958]: audit 2023-11-17T07:24:44.669105+0000 mon.smithi097 (mon.0) 1082 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:46.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:45 smithi097 bash[24958]: cluster 2023-11-17T07:24:44.374984+0000 mgr.smithi097.swvonl (mgr.14180) 1616 : cluster [DBG] pgmap v1148: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:46.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:45 smithi097 bash[24958]: cluster 2023-11-17T07:24:44.652864+0000 mgr.smithi097.swvonl (mgr.14180) 1617 : cluster [DBG] pgmap v1149: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:46.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:45 smithi097 bash[24958]: cephadm 2023-11-17T07:24:44.672158+0000 mgr.smithi097.swvonl (mgr.14180) 1618 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.mhbpno on smithi193 2023-11-17T07:24:46.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:45 smithi097 bash[24958]: cluster 2023-11-17T07:24:44.879389+0000 mon.smithi097 (mon.0) 1083 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:24:46.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:45 smithi097 bash[24958]: cluster 2023-11-17T07:24:44.879444+0000 mon.smithi097 (mon.0) 1084 : cluster [INF] Cluster is now healthy 2023-11-17T07:24:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:45 smithi193 bash[28559]: cluster 2023-11-17T07:24:44.374984+0000 mgr.smithi097.swvonl (mgr.14180) 1616 : cluster [DBG] pgmap v1148: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:24:46.629 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:45 smithi193 bash[28559]: cluster 2023-11-17T07:24:44.652864+0000 mgr.smithi097.swvonl (mgr.14180) 1617 : cluster [DBG] pgmap v1149: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:46.629 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:45 smithi193 bash[28559]: cephadm 2023-11-17T07:24:44.672158+0000 mgr.smithi097.swvonl (mgr.14180) 1618 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.mhbpno on smithi193 2023-11-17T07:24:46.629 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:45 smithi193 bash[28559]: cluster 2023-11-17T07:24:44.879389+0000 mon.smithi097 (mon.0) 1083 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:24:46.630 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:45 smithi193 bash[28559]: cluster 2023-11-17T07:24:44.879444+0000 mon.smithi097 (mon.0) 1084 : cluster [INF] Cluster is now healthy 2023-11-17T07:24:47.769 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:47.769 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:22:36.455240Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.kptlph on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:40.549876Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.pteakh on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-pteakh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.pteakh\nDeploy daemon haproxy.nfs.foo.smithi193.pteakh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.332733Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.uuwcbq on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-uuwcbq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.uuwcbq\nDeploy daemon haproxy.nfs.foo.smithi097.uuwcbq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:23:44.336316Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hvmoiy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:48.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:47 smithi097 bash[24958]: cluster 2023-11-17T07:24:46.653859+0000 mgr.smithi097.swvonl (mgr.14180) 1619 : cluster [DBG] pgmap v1150: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:48.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:47 smithi193 bash[28559]: cluster 2023-11-17T07:24:46.653859+0000 mgr.smithi097.swvonl (mgr.14180) 1619 : cluster [DBG] pgmap v1150: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:48.608 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:49.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:48 smithi097 bash[24958]: audit 2023-11-17T07:24:47.762999+0000 mgr.smithi097.swvonl (mgr.14180) 1620 : audit [DBG] from='client.15284 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:49.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:48 smithi097 bash[24958]: audit 2023-11-17T07:24:48.615255+0000 mon.smithi097 (mon.0) 1085 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.mhbpno"}]: dispatch 2023-11-17T07:24:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:48 smithi193 bash[28559]: audit 2023-11-17T07:24:47.762999+0000 mgr.smithi097.swvonl (mgr.14180) 1620 : audit [DBG] from='client.15284 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:48 smithi193 bash[28559]: audit 2023-11-17T07:24:48.615255+0000 mon.smithi097 (mon.0) 1085 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.mhbpno"}]: dispatch 2023-11-17T07:24:49.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: cephadm 2023-11-17T07:24:48.614028+0000 mgr.smithi097.swvonl (mgr.14180) 1621 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:24:50.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: cephadm 2023-11-17T07:24:48.614624+0000 mgr.smithi097.swvonl (mgr.14180) 1622 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: cephadm 2023-11-17T07:24:48.616762+0000 mgr.smithi097.swvonl (mgr.14180) 1623 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: cephadm 2023-11-17T07:24:48.620288+0000 mgr.smithi097.swvonl (mgr.14180) 1624 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.nzczvm on smithi097 2023-11-17T07:24:50.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:49 smithi097 bash[24958]: cluster 2023-11-17T07:24:48.654713+0000 mgr.smithi097.swvonl (mgr.14180) 1625 : cluster [DBG] pgmap v1151: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: cephadm 2023-11-17T07:24:48.614028+0000 mgr.smithi097.swvonl (mgr.14180) 1621 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: cephadm 2023-11-17T07:24:48.614624+0000 mgr.smithi097.swvonl (mgr.14180) 1622 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: cephadm 2023-11-17T07:24:48.616762+0000 mgr.smithi097.swvonl (mgr.14180) 1623 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:50.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mhbpno ... 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: cephadm 2023-11-17T07:24:48.620288+0000 mgr.smithi097.swvonl (mgr.14180) 1624 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.nzczvm on smithi097 2023-11-17T07:24:50.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:49 smithi193 bash[28559]: cluster 2023-11-17T07:24:48.654713+0000 mgr.smithi097.swvonl (mgr.14180) 1625 : cluster [DBG] pgmap v1151: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:52.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:51 smithi097 bash[24958]: cluster 2023-11-17T07:24:50.655718+0000 mgr.smithi097.swvonl (mgr.14180) 1626 : cluster [DBG] pgmap v1152: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:52.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:51 smithi193 bash[28559]: cluster 2023-11-17T07:24:50.655718+0000 mgr.smithi097.swvonl (mgr.14180) 1626 : cluster [DBG] pgmap v1152: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:53.673 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:53.673 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:24:54.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cluster 2023-11-17T07:24:52.656388+0000 mgr.smithi097.swvonl (mgr.14180) 1627 : cluster [DBG] pgmap v1153: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cephadm 2023-11-17T07:24:52.988622+0000 mgr.smithi097.swvonl (mgr.14180) 1628 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:24:54.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cephadm 2023-11-17T07:24:52.989155+0000 mgr.smithi097.swvonl (mgr.14180) 1629 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: audit 2023-11-17T07:24:52.989842+0000 mon.smithi097 (mon.0) 1086 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.nzczvm"}]: dispatch 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cephadm 2023-11-17T07:24:52.991098+0000 mgr.smithi097.swvonl (mgr.14180) 1630 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cephadm 2023-11-17T07:24:52.992943+0000 mgr.smithi097.swvonl (mgr.14180) 1631 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:24:54.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cephadm 2023-11-17T07:24:52.994484+0000 mgr.smithi097.swvonl (mgr.14180) 1632 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:24:54.054 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:53 smithi097 bash[24958]: cluster 2023-11-17T07:24:52.995672+0000 mgr.smithi097.swvonl (mgr.14180) 1633 : cluster [DBG] pgmap v1154: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2023-11-17T07:24:54.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cluster 2023-11-17T07:24:52.656388+0000 mgr.smithi097.swvonl (mgr.14180) 1627 : cluster [DBG] pgmap v1153: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cephadm 2023-11-17T07:24:52.988622+0000 mgr.smithi097.swvonl (mgr.14180) 1628 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cephadm 2023-11-17T07:24:52.989155+0000 mgr.smithi097.swvonl (mgr.14180) 1629 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: audit 2023-11-17T07:24:52.989842+0000 mon.smithi097 (mon.0) 1086 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.nzczvm"}]: dispatch 2023-11-17T07:24:54.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cephadm 2023-11-17T07:24:52.991098+0000 mgr.smithi097.swvonl (mgr.14180) 1630 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.nzczvm ... 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:24:54.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:24:54.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cephadm 2023-11-17T07:24:52.992943+0000 mgr.smithi097.swvonl (mgr.14180) 1631 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:24:54.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cephadm 2023-11-17T07:24:52.994484+0000 mgr.smithi097.swvonl (mgr.14180) 1632 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:24:54.311 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:53 smithi193 bash[28559]: cluster 2023-11-17T07:24:52.995672+0000 mgr.smithi097.swvonl (mgr.14180) 1633 : cluster [DBG] pgmap v1154: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2023-11-17T07:24:54.569 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:24:55.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:54 smithi193 bash[28559]: audit 2023-11-17T07:24:53.667379+0000 mgr.smithi097.swvonl (mgr.14180) 1634 : audit [DBG] from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:55.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:54 smithi193 bash[28559]: cluster 2023-11-17T07:24:53.917511+0000 mon.smithi097 (mon.0) 1087 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:24:55.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:54 smithi193 bash[28559]: audit 2023-11-17T07:24:54.580849+0000 mon.smithi097 (mon.0) 1088 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:55.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:54 smithi097 bash[24958]: audit 2023-11-17T07:24:53.667379+0000 mgr.smithi097.swvonl (mgr.14180) 1634 : audit [DBG] from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:24:55.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:54 smithi097 bash[24958]: cluster 2023-11-17T07:24:53.917511+0000 mon.smithi097 (mon.0) 1087 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:24:55.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:54 smithi097 bash[24958]: audit 2023-11-17T07:24:54.580849+0000 mon.smithi097 (mon.0) 1088 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:24:55.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:24:56.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:55 smithi097 bash[24958]: cluster 2023-11-17T07:24:54.997156+0000 mgr.smithi097.swvonl (mgr.14180) 1635 : cluster [DBG] pgmap v1155: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:24:56.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:55 smithi193 bash[28559]: cluster 2023-11-17T07:24:54.997156+0000 mgr.smithi097.swvonl (mgr.14180) 1635 : cluster [DBG] pgmap v1155: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:24:58.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:24:57 smithi097 bash[24958]: cluster 2023-11-17T07:24:56.998127+0000 mgr.smithi097.swvonl (mgr.14180) 1636 : cluster [DBG] pgmap v1156: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:24:58.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:24:57 smithi193 bash[28559]: cluster 2023-11-17T07:24:56.998127+0000 mgr.smithi097.swvonl (mgr.14180) 1636 : cluster [DBG] pgmap v1156: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:24:59.256 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:24:59.257 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:00.103 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:00 smithi097 bash[24958]: cluster 2023-11-17T07:24:58.999246+0000 mgr.smithi097.swvonl (mgr.14180) 1637 : cluster [DBG] pgmap v1157: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:00.104 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:00.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:00 smithi193 bash[28559]: cluster 2023-11-17T07:24:58.999246+0000 mgr.smithi097.swvonl (mgr.14180) 1637 : cluster [DBG] pgmap v1157: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:01.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:01.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:01 smithi097 bash[24958]: audit 2023-11-17T07:24:59.245508+0000 mgr.smithi097.swvonl (mgr.14180) 1638 : audit [DBG] from='client.15292 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:01.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:01 smithi097 bash[24958]: cluster 2023-11-17T07:25:01.000709+0000 mgr.smithi097.swvonl (mgr.14180) 1639 : cluster [DBG] pgmap v1158: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:01.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:01 smithi193 bash[28559]: audit 2023-11-17T07:24:59.245508+0000 mgr.smithi097.swvonl (mgr.14180) 1638 : audit [DBG] from='client.15292 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:01.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:01 smithi193 bash[28559]: cluster 2023-11-17T07:25:01.000709+0000 mgr.smithi097.swvonl (mgr.14180) 1639 : cluster [DBG] pgmap v1158: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:04.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:04 smithi097 bash[24958]: cluster 2023-11-17T07:25:03.001473+0000 mgr.smithi097.swvonl (mgr.14180) 1640 : cluster [DBG] pgmap v1159: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:04.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:04 smithi193 bash[28559]: cluster 2023-11-17T07:25:03.001473+0000 mgr.smithi097.swvonl (mgr.14180) 1640 : cluster [DBG] pgmap v1159: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2023-11-17T07:25:04.674 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:04.674 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:05 smithi097 bash[24958]: audit 2023-11-17T07:25:04.663924+0000 mgr.smithi097.swvonl (mgr.14180) 1641 : audit [DBG] from='client.15296 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:05 smithi097 bash[24958]: cluster 2023-11-17T07:25:05.002509+0000 mgr.smithi097.swvonl (mgr.14180) 1642 : cluster [DBG] pgmap v1160: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:05.534 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:05.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:05 smithi193 bash[28559]: audit 2023-11-17T07:25:04.663924+0000 mgr.smithi097.swvonl (mgr.14180) 1641 : audit [DBG] from='client.15296 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:05.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:05 smithi193 bash[28559]: cluster 2023-11-17T07:25:05.002509+0000 mgr.smithi097.swvonl (mgr.14180) 1642 : cluster [DBG] pgmap v1160: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:06.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:08.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:07 smithi097 bash[24958]: cluster 2023-11-17T07:25:07.003104+0000 mgr.smithi097.swvonl (mgr.14180) 1643 : cluster [DBG] pgmap v1161: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:08.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:07 smithi193 bash[28559]: cluster 2023-11-17T07:25:07.003104+0000 mgr.smithi097.swvonl (mgr.14180) 1643 : cluster [DBG] pgmap v1161: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:09.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:09 smithi097 bash[24958]: cluster 2023-11-17T07:25:09.004126+0000 mgr.smithi097.swvonl (mgr.14180) 1644 : cluster [DBG] pgmap v1162: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:09.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:09 smithi193 bash[28559]: cluster 2023-11-17T07:25:09.004126+0000 mgr.smithi097.swvonl (mgr.14180) 1644 : cluster [DBG] pgmap v1162: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:10.107 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:10.107 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:10.939 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:11 smithi097 bash[24958]: audit 2023-11-17T07:25:10.097233+0000 mgr.smithi097.swvonl (mgr.14180) 1645 : audit [DBG] from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:11 smithi097 bash[24958]: cluster 2023-11-17T07:25:11.005473+0000 mgr.smithi097.swvonl (mgr.14180) 1646 : cluster [DBG] pgmap v1163: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:11 smithi193 bash[28559]: audit 2023-11-17T07:25:10.097233+0000 mgr.smithi097.swvonl (mgr.14180) 1645 : audit [DBG] from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:11 smithi193 bash[28559]: cluster 2023-11-17T07:25:11.005473+0000 mgr.smithi097.swvonl (mgr.14180) 1646 : cluster [DBG] pgmap v1163: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:11.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:13.463 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:13 smithi097 bash[24958]: cluster 2023-11-17T07:25:13.006079+0000 mgr.smithi097.swvonl (mgr.14180) 1647 : cluster [DBG] pgmap v1164: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:13.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:13 smithi193 bash[28559]: cluster 2023-11-17T07:25:13.006079+0000 mgr.smithi097.swvonl (mgr.14180) 1647 : cluster [DBG] pgmap v1164: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:15.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:15 smithi097 bash[24958]: cluster 2023-11-17T07:25:15.007392+0000 mgr.smithi097.swvonl (mgr.14180) 1648 : cluster [DBG] pgmap v1165: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:15.383 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:15.383 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:15.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:15 smithi193 bash[28559]: cluster 2023-11-17T07:25:15.007392+0000 mgr.smithi097.swvonl (mgr.14180) 1648 : cluster [DBG] pgmap v1165: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:16.283 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:16.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:16 smithi097 bash[24958]: audit 2023-11-17T07:25:15.376983+0000 mgr.smithi097.swvonl (mgr.14180) 1649 : audit [DBG] from='client.15304 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:16.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:16 smithi193 bash[28559]: audit 2023-11-17T07:25:15.376983+0000 mgr.smithi097.swvonl (mgr.14180) 1649 : audit [DBG] from='client.15304 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:17.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:18.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:17 smithi097 bash[24958]: cluster 2023-11-17T07:25:17.008503+0000 mgr.smithi097.swvonl (mgr.14180) 1650 : cluster [DBG] pgmap v1166: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:18.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:17 smithi193 bash[28559]: cluster 2023-11-17T07:25:17.008503+0000 mgr.smithi097.swvonl (mgr.14180) 1650 : cluster [DBG] pgmap v1166: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:19.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:19 smithi097 bash[24958]: cluster 2023-11-17T07:25:19.009321+0000 mgr.smithi097.swvonl (mgr.14180) 1651 : cluster [DBG] pgmap v1167: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:19.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:19 smithi193 bash[28559]: cluster 2023-11-17T07:25:19.009321+0000 mgr.smithi097.swvonl (mgr.14180) 1651 : cluster [DBG] pgmap v1167: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:20.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:20.953 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:21.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:21 smithi097 bash[24958]: audit 2023-11-17T07:25:20.942987+0000 mgr.smithi097.swvonl (mgr.14180) 1652 : audit [DBG] from='client.15308 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:21.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:21 smithi097 bash[24958]: cluster 2023-11-17T07:25:21.010844+0000 mgr.smithi097.swvonl (mgr.14180) 1653 : cluster [DBG] pgmap v1168: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:21.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:21 smithi193 bash[28559]: audit 2023-11-17T07:25:20.942987+0000 mgr.smithi097.swvonl (mgr.14180) 1652 : audit [DBG] from='client.15308 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:21.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:21 smithi193 bash[28559]: cluster 2023-11-17T07:25:21.010844+0000 mgr.smithi097.swvonl (mgr.14180) 1653 : cluster [DBG] pgmap v1168: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:21.777 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:22.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:23.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:23 smithi097 bash[24958]: cluster 2023-11-17T07:25:23.011758+0000 mgr.smithi097.swvonl (mgr.14180) 1654 : cluster [DBG] pgmap v1169: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:23.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:23 smithi193 bash[28559]: cluster 2023-11-17T07:25:23.011758+0000 mgr.smithi097.swvonl (mgr.14180) 1654 : cluster [DBG] pgmap v1169: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:25 smithi097 bash[24958]: cluster 2023-11-17T07:25:25.013212+0000 mgr.smithi097.swvonl (mgr.14180) 1655 : cluster [DBG] pgmap v1170: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:25.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:25 smithi193 bash[28559]: cluster 2023-11-17T07:25:25.013212+0000 mgr.smithi097.swvonl (mgr.14180) 1655 : cluster [DBG] pgmap v1170: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:26.428 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:26.429 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:27.271 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:28.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:27 smithi097 bash[24958]: audit 2023-11-17T07:25:26.417481+0000 mgr.smithi097.swvonl (mgr.14180) 1656 : audit [DBG] from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:28.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:27 smithi097 bash[24958]: cluster 2023-11-17T07:25:27.014344+0000 mgr.smithi097.swvonl (mgr.14180) 1657 : cluster [DBG] pgmap v1171: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:27 smithi193 bash[28559]: audit 2023-11-17T07:25:26.417481+0000 mgr.smithi097.swvonl (mgr.14180) 1656 : audit [DBG] from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:27 smithi193 bash[28559]: cluster 2023-11-17T07:25:27.014344+0000 mgr.smithi097.swvonl (mgr.14180) 1657 : cluster [DBG] pgmap v1171: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:28.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:29.463 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:29 smithi097 bash[24958]: cluster 2023-11-17T07:25:29.015334+0000 mgr.smithi097.swvonl (mgr.14180) 1658 : cluster [DBG] pgmap v1172: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:29.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:29 smithi193 bash[28559]: cluster 2023-11-17T07:25:29.015334+0000 mgr.smithi097.swvonl (mgr.14180) 1658 : cluster [DBG] pgmap v1172: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:31.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:31 smithi097 bash[24958]: cluster 2023-11-17T07:25:31.017038+0000 mgr.smithi097.swvonl (mgr.14180) 1659 : cluster [DBG] pgmap v1173: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:31.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:31 smithi193 bash[28559]: cluster 2023-11-17T07:25:31.017038+0000 mgr.smithi097.swvonl (mgr.14180) 1659 : cluster [DBG] pgmap v1173: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:31.833 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:31.834 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:32.682 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:33.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:32 smithi097 bash[24958]: audit 2023-11-17T07:25:31.827909+0000 mgr.smithi097.swvonl (mgr.14180) 1660 : audit [DBG] from='client.15316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:33.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:32 smithi193 bash[28559]: audit 2023-11-17T07:25:31.827909+0000 mgr.smithi097.swvonl (mgr.14180) 1660 : audit [DBG] from='client.15316 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:33.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:34.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:33 smithi097 bash[24958]: cluster 2023-11-17T07:25:33.018096+0000 mgr.smithi097.swvonl (mgr.14180) 1661 : cluster [DBG] pgmap v1174: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:34.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:33 smithi193 bash[28559]: cluster 2023-11-17T07:25:33.018096+0000 mgr.smithi097.swvonl (mgr.14180) 1661 : cluster [DBG] pgmap v1174: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:35.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:35 smithi097 bash[24958]: cluster 2023-11-17T07:25:35.018953+0000 mgr.smithi097.swvonl (mgr.14180) 1662 : cluster [DBG] pgmap v1175: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:35.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:35 smithi193 bash[28559]: cluster 2023-11-17T07:25:35.018953+0000 mgr.smithi097.swvonl (mgr.14180) 1662 : cluster [DBG] pgmap v1175: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:37.160 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:37.160 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:37.959 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:37 smithi097 bash[24958]: cluster 2023-11-17T07:25:37.019939+0000 mgr.smithi097.swvonl (mgr.14180) 1663 : cluster [DBG] pgmap v1176: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:38.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:37 smithi193 bash[28559]: cluster 2023-11-17T07:25:37.019939+0000 mgr.smithi097.swvonl (mgr.14180) 1663 : cluster [DBG] pgmap v1176: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:38.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:39.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:38 smithi097 bash[24958]: audit 2023-11-17T07:25:37.147088+0000 mgr.smithi097.swvonl (mgr.14180) 1664 : audit [DBG] from='client.15320 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:39.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:38 smithi193 bash[28559]: audit 2023-11-17T07:25:37.147088+0000 mgr.smithi097.swvonl (mgr.14180) 1664 : audit [DBG] from='client.15320 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:40.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:39 smithi097 bash[24958]: cluster 2023-11-17T07:25:39.021046+0000 mgr.smithi097.swvonl (mgr.14180) 1665 : cluster [DBG] pgmap v1177: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:40.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:39 smithi193 bash[28559]: cluster 2023-11-17T07:25:39.021046+0000 mgr.smithi097.swvonl (mgr.14180) 1665 : cluster [DBG] pgmap v1177: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:41.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:41 smithi097 bash[24958]: cluster 2023-11-17T07:25:41.022603+0000 mgr.smithi097.swvonl (mgr.14180) 1666 : cluster [DBG] pgmap v1178: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:41.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:41 smithi193 bash[28559]: cluster 2023-11-17T07:25:41.022603+0000 mgr.smithi097.swvonl (mgr.14180) 1666 : cluster [DBG] pgmap v1178: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:42.374 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:42.375 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:43.223 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:43.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:43 smithi097 bash[24958]: audit 2023-11-17T07:25:42.368799+0000 mgr.smithi097.swvonl (mgr.14180) 1667 : audit [DBG] from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:43.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:43 smithi097 bash[24958]: cluster 2023-11-17T07:25:43.023632+0000 mgr.smithi097.swvonl (mgr.14180) 1668 : cluster [DBG] pgmap v1179: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:43.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:43 smithi193 bash[28559]: audit 2023-11-17T07:25:42.368799+0000 mgr.smithi097.swvonl (mgr.14180) 1667 : audit [DBG] from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:43.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:43 smithi193 bash[28559]: cluster 2023-11-17T07:25:43.023632+0000 mgr.smithi097.swvonl (mgr.14180) 1668 : cluster [DBG] pgmap v1179: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:44.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:45.511 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:45 smithi097 bash[24958]: cluster 2023-11-17T07:25:45.024920+0000 mgr.smithi097.swvonl (mgr.14180) 1669 : cluster [DBG] pgmap v1180: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:45.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:45 smithi193 bash[28559]: cluster 2023-11-17T07:25:45.024920+0000 mgr.smithi097.swvonl (mgr.14180) 1669 : cluster [DBG] pgmap v1180: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:47.944 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:47.944 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:48.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:47 smithi097 bash[24958]: cluster 2023-11-17T07:25:47.025989+0000 mgr.smithi097.swvonl (mgr.14180) 1670 : cluster [DBG] pgmap v1181: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:48.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:47 smithi193 bash[28559]: cluster 2023-11-17T07:25:47.025989+0000 mgr.smithi097.swvonl (mgr.14180) 1670 : cluster [DBG] pgmap v1181: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:48.750 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:49.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:48 smithi097 bash[24958]: audit 2023-11-17T07:25:47.938311+0000 mgr.smithi097.swvonl (mgr.14180) 1671 : audit [DBG] from='client.15328 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:49.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:48 smithi193 bash[28559]: audit 2023-11-17T07:25:47.938311+0000 mgr.smithi097.swvonl (mgr.14180) 1671 : audit [DBG] from='client.15328 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:49.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:50.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:49 smithi097 bash[24958]: cluster 2023-11-17T07:25:49.027117+0000 mgr.smithi097.swvonl (mgr.14180) 1672 : cluster [DBG] pgmap v1182: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:50.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:49 smithi193 bash[28559]: cluster 2023-11-17T07:25:49.027117+0000 mgr.smithi097.swvonl (mgr.14180) 1672 : cluster [DBG] pgmap v1182: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:51.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:51 smithi097 bash[24958]: cluster 2023-11-17T07:25:51.028075+0000 mgr.smithi097.swvonl (mgr.14180) 1673 : cluster [DBG] pgmap v1183: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:51.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:51 smithi193 bash[28559]: cluster 2023-11-17T07:25:51.028075+0000 mgr.smithi097.swvonl (mgr.14180) 1673 : cluster [DBG] pgmap v1183: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:25:53.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:53 smithi097 bash[24958]: audit 2023-11-17T07:25:52.999196+0000 mon.smithi097 (mon.0) 1089 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:25:53.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:53 smithi193 bash[28559]: audit 2023-11-17T07:25:52.999196+0000 mon.smithi097 (mon.0) 1089 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:25:53.317 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:53.317 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:23:44.338928Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.yugkqw on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:54.174 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: cluster 2023-11-17T07:25:53.028757+0000 mgr.smithi097.swvonl (mgr.14180) 1674 : cluster [DBG] pgmap v1184: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: audit 2023-11-17T07:25:53.304375+0000 mon.smithi097 (mon.0) 1090 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: audit 2023-11-17T07:25:53.305624+0000 mon.smithi097 (mon.0) 1091 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: audit 2023-11-17T07:25:53.321742+0000 mon.smithi097 (mon.0) 1092 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: audit 2023-11-17T07:25:53.330694+0000 mon.smithi097 (mon.0) 1093 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:25:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:54 smithi097 bash[24958]: audit 2023-11-17T07:25:53.343977+0000 mon.smithi097 (mon.0) 1094 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: cluster 2023-11-17T07:25:53.028757+0000 mgr.smithi097.swvonl (mgr.14180) 1674 : cluster [DBG] pgmap v1184: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: audit 2023-11-17T07:25:53.304375+0000 mon.smithi097 (mon.0) 1090 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: audit 2023-11-17T07:25:53.305624+0000 mon.smithi097 (mon.0) 1091 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: audit 2023-11-17T07:25:53.321742+0000 mon.smithi097 (mon.0) 1092 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: audit 2023-11-17T07:25:53.330694+0000 mon.smithi097 (mon.0) 1093 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:25:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:54 smithi193 bash[28559]: audit 2023-11-17T07:25:53.343977+0000 mon.smithi097 (mon.0) 1094 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:55.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:25:55.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:55 smithi097 bash[24958]: audit 2023-11-17T07:25:53.307397+0000 mgr.smithi097.swvonl (mgr.14180) 1675 : audit [DBG] from='client.15332 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:55.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:55 smithi097 bash[24958]: cluster 2023-11-17T07:25:53.322585+0000 mgr.smithi097.swvonl (mgr.14180) 1676 : cluster [DBG] pgmap v1185: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:55.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:55 smithi097 bash[24958]: cephadm 2023-11-17T07:25:53.347938+0000 mgr.smithi097.swvonl (mgr.14180) 1677 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.csbrjj on smithi193 2023-11-17T07:25:55.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:55 smithi097 bash[24958]: cluster 2023-11-17T07:25:54.319559+0000 mon.smithi097 (mon.0) 1095 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:25:55.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:55 smithi097 bash[24958]: cluster 2023-11-17T07:25:54.319619+0000 mon.smithi097 (mon.0) 1096 : cluster [INF] Cluster is now healthy 2023-11-17T07:25:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:55 smithi193 bash[28559]: audit 2023-11-17T07:25:53.307397+0000 mgr.smithi097.swvonl (mgr.14180) 1675 : audit [DBG] from='client.15332 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:55 smithi193 bash[28559]: cluster 2023-11-17T07:25:53.322585+0000 mgr.smithi097.swvonl (mgr.14180) 1676 : cluster [DBG] pgmap v1185: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:55 smithi193 bash[28559]: cephadm 2023-11-17T07:25:53.347938+0000 mgr.smithi097.swvonl (mgr.14180) 1677 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.csbrjj on smithi193 2023-11-17T07:25:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:55 smithi193 bash[28559]: cluster 2023-11-17T07:25:54.319559+0000 mon.smithi097 (mon.0) 1095 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:25:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:55 smithi193 bash[28559]: cluster 2023-11-17T07:25:54.319619+0000 mon.smithi097 (mon.0) 1096 : cluster [INF] Cluster is now healthy 2023-11-17T07:25:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:57 smithi097 bash[24958]: cluster 2023-11-17T07:25:55.323142+0000 mgr.smithi097.swvonl (mgr.14180) 1678 : cluster [DBG] pgmap v1186: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:57 smithi097 bash[24958]: audit 2023-11-17T07:25:56.559415+0000 mon.smithi097 (mon.0) 1097 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:57 smithi193 bash[28559]: cluster 2023-11-17T07:25:55.323142+0000 mgr.smithi097.swvonl (mgr.14180) 1678 : cluster [DBG] pgmap v1186: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:57 smithi193 bash[28559]: audit 2023-11-17T07:25:56.559415+0000 mon.smithi097 (mon.0) 1097 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:25:58.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: cephadm 2023-11-17T07:25:57.103548+0000 mgr.smithi097.swvonl (mgr.14180) 1679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: cephadm 2023-11-17T07:25:57.104108+0000 mgr.smithi097.swvonl (mgr.14180) 1680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: audit 2023-11-17T07:25:57.104677+0000 mon.smithi097 (mon.0) 1098 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.csbrjj"}]: dispatch 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: cephadm 2023-11-17T07:25:57.105998+0000 mgr.smithi097.swvonl (mgr.14180) 1681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: cephadm 2023-11-17T07:25:57.110964+0000 mgr.smithi097.swvonl (mgr.14180) 1682 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.mrltoa on smithi097 2023-11-17T07:25:58.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:58 smithi097 bash[24958]: cluster 2023-11-17T07:25:57.323839+0000 mgr.smithi097.swvonl (mgr.14180) 1683 : cluster [DBG] pgmap v1187: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:58.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: cephadm 2023-11-17T07:25:57.103548+0000 mgr.smithi097.swvonl (mgr.14180) 1679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:25:58.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: cephadm 2023-11-17T07:25:57.104108+0000 mgr.smithi097.swvonl (mgr.14180) 1680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: audit 2023-11-17T07:25:57.104677+0000 mon.smithi097 (mon.0) 1098 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.csbrjj"}]: dispatch 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: cephadm 2023-11-17T07:25:57.105998+0000 mgr.smithi097.swvonl (mgr.14180) 1681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj 2023-11-17T07:25:58.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.csbrjj ... 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: cephadm 2023-11-17T07:25:57.110964+0000 mgr.smithi097.swvonl (mgr.14180) 1682 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.mrltoa on smithi097 2023-11-17T07:25:58.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:58 smithi193 bash[28559]: cluster 2023-11-17T07:25:57.323839+0000 mgr.smithi097.swvonl (mgr.14180) 1683 : cluster [DBG] pgmap v1187: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:25:58.974 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:25:58.975 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:48.616545Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mhbpno on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mhbpno\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mhbpno\nDeploy daemon haproxy.nfs.foo.smithi193.mhbpno ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.990947Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.nzczvm on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-nzczvm\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.nzczvm\nDeploy daemon haproxy.nfs.foo.smithi097.nzczvm ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:24:52.992838Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ekspde on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:25:59.257 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:25:59 smithi097 bash[24958]: audit 2023-11-17T07:25:58.964285+0000 mgr.smithi097.swvonl (mgr.14180) 1684 : audit [DBG] from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:25:59 smithi193 bash[28559]: audit 2023-11-17T07:25:58.964285+0000 mgr.smithi097.swvonl (mgr.14180) 1684 : audit [DBG] from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:25:59.862 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:00.475 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:00 smithi097 bash[24958]: cluster 2023-11-17T07:25:59.324841+0000 mgr.smithi097.swvonl (mgr.14180) 1685 : cluster [DBG] pgmap v1188: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:26:00.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:00 smithi193 bash[28559]: cluster 2023-11-17T07:25:59.324841+0000 mgr.smithi097.swvonl (mgr.14180) 1685 : cluster [DBG] pgmap v1188: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:26:00.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:01.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cephadm 2023-11-17T07:26:00.897888+0000 mgr.smithi097.swvonl (mgr.14180) 1686 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.606 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:26:33.607 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.608 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cephadm 2023-11-17T07:26:00.898160+0000 mgr.smithi097.swvonl (mgr.14180) 1687 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: audit 2023-11-17T07:26:00.898589+0000 mon.smithi097 (mon.0) 1099 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.mrltoa"}]: dispatch 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cephadm 2023-11-17T07:26:00.899930+0000 mgr.smithi097.swvonl (mgr.14180) 1688 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.609 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cephadm 2023-11-17T07:26:00.904244+0000 mgr.smithi097.swvonl (mgr.14180) 1689 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cephadm 2023-11-17T07:26:00.906199+0000 mgr.smithi097.swvonl (mgr.14180) 1690 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:26:33.610 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:01 smithi097 bash[24958]: cluster 2023-11-17T07:26:00.907646+0000 mgr.smithi097.swvonl (mgr.14180) 1691 : cluster [DBG] pgmap v1189: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:26:33.612 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:02 smithi097 bash[24958]: cluster 2023-11-17T07:26:01.085241+0000 mon.smithi097 (mon.0) 1100 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:26:33.612 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:03 smithi097 bash[24958]: cluster 2023-11-17T07:26:02.908597+0000 mgr.smithi097.swvonl (mgr.14180) 1692 : cluster [DBG] pgmap v1190: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:26:33.613 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:26:33.614 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:26:33.614 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cephadm 2023-11-17T07:26:00.897888+0000 mgr.smithi097.swvonl (mgr.14180) 1686 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.616 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.617 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cephadm 2023-11-17T07:26:00.898160+0000 mgr.smithi097.swvonl (mgr.14180) 1687 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: audit 2023-11-17T07:26:00.898589+0000 mon.smithi097 (mon.0) 1099 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.mrltoa"}]: dispatch 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cephadm 2023-11-17T07:26:00.899930+0000 mgr.smithi097.swvonl (mgr.14180) 1688 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.618 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mrltoa ... 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cephadm 2023-11-17T07:26:00.904244+0000 mgr.smithi097.swvonl (mgr.14180) 1689 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cephadm 2023-11-17T07:26:00.906199+0000 mgr.smithi097.swvonl (mgr.14180) 1690 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:01 smithi193 bash[28559]: cluster 2023-11-17T07:26:00.907646+0000 mgr.smithi097.swvonl (mgr.14180) 1691 : cluster [DBG] pgmap v1189: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:26:33.619 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:02 smithi193 bash[28559]: cluster 2023-11-17T07:26:01.085241+0000 mon.smithi097 (mon.0) 1100 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:03 smithi193 bash[28559]: cluster 2023-11-17T07:26:02.908597+0000 mgr.smithi097.swvonl (mgr.14180) 1692 : cluster [DBG] pgmap v1190: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:05 smithi193 bash[28559]: audit 2023-11-17T07:26:04.628200+0000 mon.smithi097 (mon.0) 1101 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:05 smithi193 bash[28559]: audit 2023-11-17T07:26:04.778627+0000 mgr.smithi097.swvonl (mgr.14180) 1693 : audit [DBG] from='client.15340 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:05 smithi193 bash[28559]: cluster 2023-11-17T07:26:04.909984+0000 mgr.smithi097.swvonl (mgr.14180) 1694 : cluster [DBG] pgmap v1191: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:07 smithi193 bash[28559]: cluster 2023-11-17T07:26:06.911031+0000 mgr.smithi097.swvonl (mgr.14180) 1695 : cluster [DBG] pgmap v1192: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:09 smithi193 bash[28559]: cluster 2023-11-17T07:26:08.912108+0000 mgr.smithi097.swvonl (mgr.14180) 1696 : cluster [DBG] pgmap v1193: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:11 smithi193 bash[28559]: cluster 2023-11-17T07:26:10.913368+0000 mgr.smithi097.swvonl (mgr.14180) 1697 : cluster [DBG] pgmap v1194: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:11 smithi193 bash[28559]: audit 2023-11-17T07:26:11.558580+0000 mon.smithi097 (mon.0) 1102 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:13 smithi193 bash[28559]: cluster 2023-11-17T07:26:12.914405+0000 mgr.smithi097.swvonl (mgr.14180) 1698 : cluster [DBG] pgmap v1195: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:15 smithi193 bash[28559]: cluster 2023-11-17T07:26:14.915646+0000 mgr.smithi097.swvonl (mgr.14180) 1699 : cluster [DBG] pgmap v1196: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:17 smithi193 bash[28559]: cluster 2023-11-17T07:26:16.916680+0000 mgr.smithi097.swvonl (mgr.14180) 1700 : cluster [DBG] pgmap v1197: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:19 smithi193 bash[28559]: cluster 2023-11-17T07:26:18.917773+0000 mgr.smithi097.swvonl (mgr.14180) 1701 : cluster [DBG] pgmap v1198: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:21 smithi193 bash[28559]: cluster 2023-11-17T07:26:20.919169+0000 mgr.smithi097.swvonl (mgr.14180) 1702 : cluster [DBG] pgmap v1199: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:33.620 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:23 smithi193 bash[28559]: cluster 2023-11-17T07:26:22.920126+0000 mgr.smithi097.swvonl (mgr.14180) 1703 : cluster [DBG] pgmap v1200: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:25 smithi193 bash[28559]: cluster 2023-11-17T07:26:24.921375+0000 mgr.smithi097.swvonl (mgr.14180) 1704 : cluster [DBG] pgmap v1201: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:27 smithi193 bash[28559]: cluster 2023-11-17T07:26:26.922447+0000 mgr.smithi097.swvonl (mgr.14180) 1705 : cluster [DBG] pgmap v1202: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:29 smithi193 bash[28559]: cluster 2023-11-17T07:26:28.923442+0000 mgr.smithi097.swvonl (mgr.14180) 1706 : cluster [DBG] pgmap v1203: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:31 smithi193 bash[28559]: cluster 2023-11-17T07:26:30.924679+0000 mgr.smithi097.swvonl (mgr.14180) 1707 : cluster [DBG] pgmap v1204: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:05 smithi097 bash[24958]: audit 2023-11-17T07:26:04.628200+0000 mon.smithi097 (mon.0) 1101 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:05 smithi097 bash[24958]: audit 2023-11-17T07:26:04.778627+0000 mgr.smithi097.swvonl (mgr.14180) 1693 : audit [DBG] from='client.15340 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:33.621 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:05 smithi097 bash[24958]: cluster 2023-11-17T07:26:04.909984+0000 mgr.smithi097.swvonl (mgr.14180) 1694 : cluster [DBG] pgmap v1191: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:07 smithi097 bash[24958]: cluster 2023-11-17T07:26:06.911031+0000 mgr.smithi097.swvonl (mgr.14180) 1695 : cluster [DBG] pgmap v1192: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:09 smithi097 bash[24958]: cluster 2023-11-17T07:26:08.912108+0000 mgr.smithi097.swvonl (mgr.14180) 1696 : cluster [DBG] pgmap v1193: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:11 smithi097 bash[24958]: cluster 2023-11-17T07:26:10.913368+0000 mgr.smithi097.swvonl (mgr.14180) 1697 : cluster [DBG] pgmap v1194: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:11 smithi097 bash[24958]: audit 2023-11-17T07:26:11.558580+0000 mon.smithi097 (mon.0) 1102 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:13 smithi097 bash[24958]: cluster 2023-11-17T07:26:12.914405+0000 mgr.smithi097.swvonl (mgr.14180) 1698 : cluster [DBG] pgmap v1195: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:15 smithi097 bash[24958]: cluster 2023-11-17T07:26:14.915646+0000 mgr.smithi097.swvonl (mgr.14180) 1699 : cluster [DBG] pgmap v1196: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:17 smithi097 bash[24958]: cluster 2023-11-17T07:26:16.916680+0000 mgr.smithi097.swvonl (mgr.14180) 1700 : cluster [DBG] pgmap v1197: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:19 smithi097 bash[24958]: cluster 2023-11-17T07:26:18.917773+0000 mgr.smithi097.swvonl (mgr.14180) 1701 : cluster [DBG] pgmap v1198: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:21 smithi097 bash[24958]: cluster 2023-11-17T07:26:20.919169+0000 mgr.smithi097.swvonl (mgr.14180) 1702 : cluster [DBG] pgmap v1199: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:23 smithi097 bash[24958]: cluster 2023-11-17T07:26:22.920126+0000 mgr.smithi097.swvonl (mgr.14180) 1703 : cluster [DBG] pgmap v1200: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:25 smithi097 bash[24958]: cluster 2023-11-17T07:26:24.921375+0000 mgr.smithi097.swvonl (mgr.14180) 1704 : cluster [DBG] pgmap v1201: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:27 smithi097 bash[24958]: cluster 2023-11-17T07:26:26.922447+0000 mgr.smithi097.swvonl (mgr.14180) 1705 : cluster [DBG] pgmap v1202: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.622 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:29 smithi097 bash[24958]: cluster 2023-11-17T07:26:28.923442+0000 mgr.smithi097.swvonl (mgr.14180) 1706 : cluster [DBG] pgmap v1203: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:33.623 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:31 smithi097 bash[24958]: cluster 2023-11-17T07:26:30.924679+0000 mgr.smithi097.swvonl (mgr.14180) 1707 : cluster [DBG] pgmap v1204: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:34.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:33 smithi097 bash[24958]: cluster 2023-11-17T07:26:32.925638+0000 mgr.smithi097.swvonl (mgr.14180) 1708 : cluster [DBG] pgmap v1205: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:34.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:33 smithi193 bash[28559]: cluster 2023-11-17T07:26:32.925638+0000 mgr.smithi097.swvonl (mgr.14180) 1708 : cluster [DBG] pgmap v1205: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:34.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:36.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:35 smithi097 bash[24958]: cluster 2023-11-17T07:26:34.927007+0000 mgr.smithi097.swvonl (mgr.14180) 1709 : cluster [DBG] pgmap v1206: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:35 smithi193 bash[28559]: cluster 2023-11-17T07:26:34.927007+0000 mgr.smithi097.swvonl (mgr.14180) 1709 : cluster [DBG] pgmap v1206: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:38.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:37 smithi097 bash[24958]: cluster 2023-11-17T07:26:36.927949+0000 mgr.smithi097.swvonl (mgr.14180) 1710 : cluster [DBG] pgmap v1207: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:38.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:37 smithi193 bash[28559]: cluster 2023-11-17T07:26:36.927949+0000 mgr.smithi097.swvonl (mgr.14180) 1710 : cluster [DBG] pgmap v1207: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:38.377 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:26:38.377 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:26:39.258 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:40.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:40.270 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:39 smithi097 bash[24958]: audit 2023-11-17T07:26:38.366492+0000 mgr.smithi097.swvonl (mgr.14180) 1711 : audit [DBG] from='client.15344 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:40.270 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:39 smithi097 bash[24958]: cluster 2023-11-17T07:26:38.928743+0000 mgr.smithi097.swvonl (mgr.14180) 1712 : cluster [DBG] pgmap v1208: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:39 smithi193 bash[28559]: audit 2023-11-17T07:26:38.366492+0000 mgr.smithi097.swvonl (mgr.14180) 1711 : audit [DBG] from='client.15344 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:39 smithi193 bash[28559]: cluster 2023-11-17T07:26:38.928743+0000 mgr.smithi097.swvonl (mgr.14180) 1712 : cluster [DBG] pgmap v1208: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:42.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:41 smithi097 bash[24958]: cluster 2023-11-17T07:26:40.930060+0000 mgr.smithi097.swvonl (mgr.14180) 1713 : cluster [DBG] pgmap v1209: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:42.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:41 smithi193 bash[28559]: cluster 2023-11-17T07:26:40.930060+0000 mgr.smithi097.swvonl (mgr.14180) 1713 : cluster [DBG] pgmap v1209: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:43.778 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:26:43.779 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:26:44.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:43 smithi097 bash[24958]: cluster 2023-11-17T07:26:42.931124+0000 mgr.smithi097.swvonl (mgr.14180) 1714 : cluster [DBG] pgmap v1210: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:44.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:43 smithi193 bash[28559]: cluster 2023-11-17T07:26:42.931124+0000 mgr.smithi097.swvonl (mgr.14180) 1714 : cluster [DBG] pgmap v1210: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:44.665 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:45.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:45 smithi193 bash[28559]: audit 2023-11-17T07:26:43.768802+0000 mgr.smithi097.swvonl (mgr.14180) 1715 : audit [DBG] from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:45 smithi097 bash[24958]: audit 2023-11-17T07:26:43.768802+0000 mgr.smithi097.swvonl (mgr.14180) 1715 : audit [DBG] from='client.15348 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:45.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:46.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:46 smithi097 bash[24958]: cluster 2023-11-17T07:26:44.932603+0000 mgr.smithi097.swvonl (mgr.14180) 1716 : cluster [DBG] pgmap v1211: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:46 smithi193 bash[28559]: cluster 2023-11-17T07:26:44.932603+0000 mgr.smithi097.swvonl (mgr.14180) 1716 : cluster [DBG] pgmap v1211: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:48.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:47 smithi097 bash[24958]: cluster 2023-11-17T07:26:46.933687+0000 mgr.smithi097.swvonl (mgr.14180) 1717 : cluster [DBG] pgmap v1212: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:48.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:47 smithi193 bash[28559]: cluster 2023-11-17T07:26:46.933687+0000 mgr.smithi097.swvonl (mgr.14180) 1717 : cluster [DBG] pgmap v1212: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:49.655 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:26:49.656 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:26:50.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:49 smithi097 bash[24958]: cluster 2023-11-17T07:26:48.934747+0000 mgr.smithi097.swvonl (mgr.14180) 1718 : cluster [DBG] pgmap v1213: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:50.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:49 smithi193 bash[28559]: cluster 2023-11-17T07:26:48.934747+0000 mgr.smithi097.swvonl (mgr.14180) 1718 : cluster [DBG] pgmap v1213: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:50.553 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:51.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:51 smithi097 bash[24958]: audit 2023-11-17T07:26:49.643249+0000 mgr.smithi097.swvonl (mgr.14180) 1719 : audit [DBG] from='client.15352 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:51.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:51 smithi193 bash[28559]: audit 2023-11-17T07:26:49.643249+0000 mgr.smithi097.swvonl (mgr.14180) 1719 : audit [DBG] from='client.15352 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:51.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:52.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:52 smithi097 bash[24958]: cluster 2023-11-17T07:26:50.936120+0000 mgr.smithi097.swvonl (mgr.14180) 1720 : cluster [DBG] pgmap v1214: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:52.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:52 smithi193 bash[28559]: cluster 2023-11-17T07:26:50.936120+0000 mgr.smithi097.swvonl (mgr.14180) 1720 : cluster [DBG] pgmap v1214: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:26:54.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:54 smithi097 bash[24958]: cluster 2023-11-17T07:26:52.936807+0000 mgr.smithi097.swvonl (mgr.14180) 1721 : cluster [DBG] pgmap v1215: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:54.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:54 smithi193 bash[28559]: cluster 2023-11-17T07:26:52.936807+0000 mgr.smithi097.swvonl (mgr.14180) 1721 : cluster [DBG] pgmap v1215: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:55.373 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:26:55.373 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:26:56.238 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:26:56.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:56 smithi097 bash[24958]: cluster 2023-11-17T07:26:54.937729+0000 mgr.smithi097.swvonl (mgr.14180) 1722 : cluster [DBG] pgmap v1216: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:56.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:56 smithi193 bash[28559]: cluster 2023-11-17T07:26:54.937729+0000 mgr.smithi097.swvonl (mgr.14180) 1722 : cluster [DBG] pgmap v1216: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:57.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:26:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:57 smithi097 bash[24958]: audit 2023-11-17T07:26:55.367231+0000 mgr.smithi097.swvonl (mgr.14180) 1723 : audit [DBG] from='client.15356 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:57 smithi193 bash[28559]: audit 2023-11-17T07:26:55.367231+0000 mgr.smithi097.swvonl (mgr.14180) 1723 : audit [DBG] from='client.15356 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:26:58.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:26:58 smithi097 bash[24958]: cluster 2023-11-17T07:26:56.938805+0000 mgr.smithi097.swvonl (mgr.14180) 1724 : cluster [DBG] pgmap v1217: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:26:58.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:26:58 smithi193 bash[28559]: cluster 2023-11-17T07:26:56.938805+0000 mgr.smithi097.swvonl (mgr.14180) 1724 : cluster [DBG] pgmap v1217: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:00.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:00 smithi097 bash[24958]: cluster 2023-11-17T07:26:58.939555+0000 mgr.smithi097.swvonl (mgr.14180) 1725 : cluster [DBG] pgmap v1218: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:00.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:00 smithi193 bash[28559]: cluster 2023-11-17T07:26:58.939555+0000 mgr.smithi097.swvonl (mgr.14180) 1725 : cluster [DBG] pgmap v1218: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:00.875 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:00.875 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:24:52.994373Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.fimgom on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:01.156 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:01 smithi097 bash[24958]: audit 2023-11-17T07:27:00.910632+0000 mon.smithi097 (mon.0) 1103 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:27:01.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:01 smithi193 bash[28559]: audit 2023-11-17T07:27:00.910632+0000 mon.smithi097 (mon.0) 1103 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:27:01.761 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:00.866899+0000 mgr.smithi097.swvonl (mgr.14180) 1726 : audit [DBG] from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: cluster 2023-11-17T07:27:00.940303+0000 mgr.smithi097.swvonl (mgr.14180) 1727 : cluster [DBG] pgmap v1219: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:01.219384+0000 mon.smithi097 (mon.0) 1104 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:27:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:01.220881+0000 mon.smithi097 (mon.0) 1105 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:27:02.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:01.228108+0000 mon.smithi097 (mon.0) 1106 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:02.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:01.233270+0000 mon.smithi097 (mon.0) 1107 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:27:02.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:02 smithi097 bash[24958]: audit 2023-11-17T07:27:01.246498+0000 mon.smithi097 (mon.0) 1108 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:00.866899+0000 mgr.smithi097.swvonl (mgr.14180) 1726 : audit [DBG] from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: cluster 2023-11-17T07:27:00.940303+0000 mgr.smithi097.swvonl (mgr.14180) 1727 : cluster [DBG] pgmap v1219: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:01.219384+0000 mon.smithi097 (mon.0) 1104 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:01.220881+0000 mon.smithi097 (mon.0) 1105 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:01.228108+0000 mon.smithi097 (mon.0) 1106 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:01.233270+0000 mon.smithi097 (mon.0) 1107 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:27:02.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:02 smithi193 bash[28559]: audit 2023-11-17T07:27:01.246498+0000 mon.smithi097 (mon.0) 1108 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:02.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:03 smithi097 bash[24958]: cluster 2023-11-17T07:27:01.228731+0000 mgr.smithi097.swvonl (mgr.14180) 1728 : cluster [DBG] pgmap v1220: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:03 smithi097 bash[24958]: cephadm 2023-11-17T07:27:01.249125+0000 mgr.smithi097.swvonl (mgr.14180) 1729 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.iclnsq on smithi193 2023-11-17T07:27:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:03 smithi097 bash[24958]: cluster 2023-11-17T07:27:02.226063+0000 mon.smithi097 (mon.0) 1109 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:27:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:03 smithi097 bash[24958]: cluster 2023-11-17T07:27:02.226119+0000 mon.smithi097 (mon.0) 1110 : cluster [INF] Cluster is now healthy 2023-11-17T07:27:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:03 smithi193 bash[28559]: cluster 2023-11-17T07:27:01.228731+0000 mgr.smithi097.swvonl (mgr.14180) 1728 : cluster [DBG] pgmap v1220: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:03 smithi193 bash[28559]: cephadm 2023-11-17T07:27:01.249125+0000 mgr.smithi097.swvonl (mgr.14180) 1729 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.iclnsq on smithi193 2023-11-17T07:27:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:03 smithi193 bash[28559]: cluster 2023-11-17T07:27:02.226063+0000 mon.smithi097 (mon.0) 1109 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:27:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:03 smithi193 bash[28559]: cluster 2023-11-17T07:27:02.226119+0000 mon.smithi097 (mon.0) 1110 : cluster [INF] Cluster is now healthy 2023-11-17T07:27:05.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:05 smithi193 bash[28559]: cluster 2023-11-17T07:27:03.229309+0000 mgr.smithi097.swvonl (mgr.14180) 1730 : cluster [DBG] pgmap v1221: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:05.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:05 smithi097 bash[24958]: cluster 2023-11-17T07:27:03.229309+0000 mgr.smithi097.swvonl (mgr.14180) 1730 : cluster [DBG] pgmap v1221: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:06.363 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:06 smithi097 bash[24958]: audit 2023-11-17T07:27:05.174502+0000 mon.smithi097 (mon.0) 1111 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.iclnsq"}]: dispatch 2023-11-17T07:27:06.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:06 smithi193 bash[28559]: audit 2023-11-17T07:27:05.174502+0000 mon.smithi097 (mon.0) 1111 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.iclnsq"}]: dispatch 2023-11-17T07:27:06.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:06.732 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:25:57.105704Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.csbrjj on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-csbrjj\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.csbrjj\nDeploy daemon haproxy.nfs.foo.smithi193.csbrjj ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.899638Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mrltoa on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mrltoa\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mrltoa\nDeploy daemon haproxy.nfs.foo.smithi097.mrltoa ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:26:00.904114Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.shrwkb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: cephadm 2023-11-17T07:27:05.173470+0000 mgr.smithi097.swvonl (mgr.14180) 1731 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: cephadm 2023-11-17T07:27:05.173934+0000 mgr.smithi097.swvonl (mgr.14180) 1732 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: cephadm 2023-11-17T07:27:05.175827+0000 mgr.smithi097.swvonl (mgr.14180) 1733 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: cephadm 2023-11-17T07:27:05.178021+0000 mgr.smithi097.swvonl (mgr.14180) 1734 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ujsimt on smithi097 2023-11-17T07:27:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:07 smithi097 bash[24958]: cluster 2023-11-17T07:27:05.229921+0000 mgr.smithi097.swvonl (mgr.14180) 1735 : cluster [DBG] pgmap v1222: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: cephadm 2023-11-17T07:27:05.173470+0000 mgr.smithi097.swvonl (mgr.14180) 1731 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: cephadm 2023-11-17T07:27:05.173934+0000 mgr.smithi097.swvonl (mgr.14180) 1732 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: cephadm 2023-11-17T07:27:05.175827+0000 mgr.smithi097.swvonl (mgr.14180) 1733 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iclnsq ... 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: cephadm 2023-11-17T07:27:05.178021+0000 mgr.smithi097.swvonl (mgr.14180) 1734 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ujsimt on smithi097 2023-11-17T07:27:07.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:07 smithi193 bash[28559]: cluster 2023-11-17T07:27:05.229921+0000 mgr.smithi097.swvonl (mgr.14180) 1735 : cluster [DBG] pgmap v1222: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:07.569 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:08.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:08 smithi097 bash[24958]: audit 2023-11-17T07:27:06.721786+0000 mgr.smithi097.swvonl (mgr.14180) 1736 : audit [DBG] from='client.15364 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:08.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:08 smithi193 bash[28559]: audit 2023-11-17T07:27:06.721786+0000 mgr.smithi097.swvonl (mgr.14180) 1736 : audit [DBG] from='client.15364 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:08.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:09.501 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:09 smithi097 bash[24958]: cluster 2023-11-17T07:27:07.230553+0000 mgr.smithi097.swvonl (mgr.14180) 1737 : cluster [DBG] pgmap v1223: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:09.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:09 smithi193 bash[28559]: cluster 2023-11-17T07:27:07.230553+0000 mgr.smithi097.swvonl (mgr.14180) 1737 : cluster [DBG] pgmap v1223: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:10.508 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:10 smithi097 bash[24958]: audit 2023-11-17T07:27:09.777819+0000 mon.smithi097 (mon.0) 1112 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ujsimt"}]: dispatch 2023-11-17T07:27:10.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:10 smithi193 bash[28559]: audit 2023-11-17T07:27:09.777819+0000 mon.smithi097 (mon.0) 1112 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ujsimt"}]: dispatch 2023-11-17T07:27:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cluster 2023-11-17T07:27:09.231303+0000 mgr.smithi097.swvonl (mgr.14180) 1738 : cluster [DBG] pgmap v1224: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cephadm 2023-11-17T07:27:09.776848+0000 mgr.smithi097.swvonl (mgr.14180) 1739 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cephadm 2023-11-17T07:27:09.777289+0000 mgr.smithi097.swvonl (mgr.14180) 1740 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cephadm 2023-11-17T07:27:09.779075+0000 mgr.smithi097.swvonl (mgr.14180) 1741 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cephadm 2023-11-17T07:27:09.781732+0000 mgr.smithi097.swvonl (mgr.14180) 1742 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cephadm 2023-11-17T07:27:09.784107+0000 mgr.smithi097.swvonl (mgr.14180) 1743 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cluster 2023-11-17T07:27:09.785522+0000 mgr.smithi097.swvonl (mgr.14180) 1744 : cluster [DBG] pgmap v1225: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2023-11-17T07:27:11.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:11 smithi097 bash[24958]: cluster 2023-11-17T07:27:10.071585+0000 mon.smithi097 (mon.0) 1113 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:27:11.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cluster 2023-11-17T07:27:09.231303+0000 mgr.smithi097.swvonl (mgr.14180) 1738 : cluster [DBG] pgmap v1224: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:27:11.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cephadm 2023-11-17T07:27:09.776848+0000 mgr.smithi097.swvonl (mgr.14180) 1739 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cephadm 2023-11-17T07:27:09.777289+0000 mgr.smithi097.swvonl (mgr.14180) 1740 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cephadm 2023-11-17T07:27:09.779075+0000 mgr.smithi097.swvonl (mgr.14180) 1741 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:27:11.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ujsimt ... 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:27:11.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cephadm 2023-11-17T07:27:09.781732+0000 mgr.smithi097.swvonl (mgr.14180) 1742 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:27:11.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cephadm 2023-11-17T07:27:09.784107+0000 mgr.smithi097.swvonl (mgr.14180) 1743 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:27:11.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cluster 2023-11-17T07:27:09.785522+0000 mgr.smithi097.swvonl (mgr.14180) 1744 : cluster [DBG] pgmap v1225: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2023-11-17T07:27:11.563 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:11 smithi193 bash[28559]: cluster 2023-11-17T07:27:10.071585+0000 mon.smithi097 (mon.0) 1113 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:27:13.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:12 smithi097 bash[24958]: cluster 2023-11-17T07:27:11.786432+0000 mgr.smithi097.swvonl (mgr.14180) 1745 : cluster [DBG] pgmap v1226: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:13.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:12 smithi193 bash[28559]: cluster 2023-11-17T07:27:11.786432+0000 mgr.smithi097.swvonl (mgr.14180) 1745 : cluster [DBG] pgmap v1226: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:13.084 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:13.085 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:13.910 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:14.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:13 smithi097 bash[24958]: audit 2023-11-17T07:27:13.073561+0000 mgr.smithi097.swvonl (mgr.14180) 1746 : audit [DBG] from='client.15368 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:14.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:13 smithi193 bash[28559]: audit 2023-11-17T07:27:13.073561+0000 mgr.smithi097.swvonl (mgr.14180) 1746 : audit [DBG] from='client.15368 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:14.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:14 smithi193 bash[28559]: cluster 2023-11-17T07:27:13.787422+0000 mgr.smithi097.swvonl (mgr.14180) 1747 : cluster [DBG] pgmap v1227: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:14 smithi193 bash[28559]: audit 2023-11-17T07:27:14.680372+0000 mon.smithi097 (mon.0) 1114 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:15.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:14 smithi097 bash[24958]: cluster 2023-11-17T07:27:13.787422+0000 mgr.smithi097.swvonl (mgr.14180) 1747 : cluster [DBG] pgmap v1227: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:15.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:14 smithi097 bash[24958]: audit 2023-11-17T07:27:14.680372+0000 mon.smithi097 (mon.0) 1114 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:27:17.101 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:16 smithi097 bash[24958]: cluster 2023-11-17T07:27:15.788628+0000 mgr.smithi097.swvonl (mgr.14180) 1748 : cluster [DBG] pgmap v1228: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:17.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:16 smithi193 bash[28559]: cluster 2023-11-17T07:27:15.788628+0000 mgr.smithi097.swvonl (mgr.14180) 1748 : cluster [DBG] pgmap v1228: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:18.554 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:19.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:18 smithi097 bash[24958]: cluster 2023-11-17T07:27:17.789293+0000 mgr.smithi097.swvonl (mgr.14180) 1749 : cluster [DBG] pgmap v1229: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:19.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:18 smithi193 bash[28559]: cluster 2023-11-17T07:27:17.789293+0000 mgr.smithi097.swvonl (mgr.14180) 1749 : cluster [DBG] pgmap v1229: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:27:19.415 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:20.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:19 smithi097 bash[24958]: audit 2023-11-17T07:27:18.540789+0000 mgr.smithi097.swvonl (mgr.14180) 1750 : audit [DBG] from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:19 smithi193 bash[28559]: audit 2023-11-17T07:27:18.540789+0000 mgr.smithi097.swvonl (mgr.14180) 1750 : audit [DBG] from='client.15372 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:20.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:21.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:20 smithi097 bash[24958]: cluster 2023-11-17T07:27:19.790710+0000 mgr.smithi097.swvonl (mgr.14180) 1751 : cluster [DBG] pgmap v1230: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 387 B/s rd, 0 op/s 2023-11-17T07:27:21.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:20 smithi193 bash[28559]: cluster 2023-11-17T07:27:19.790710+0000 mgr.smithi097.swvonl (mgr.14180) 1751 : cluster [DBG] pgmap v1230: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 387 B/s rd, 0 op/s 2023-11-17T07:27:23.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:22 smithi097 bash[24958]: cluster 2023-11-17T07:27:21.791715+0000 mgr.smithi097.swvonl (mgr.14180) 1752 : cluster [DBG] pgmap v1231: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:23.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:22 smithi193 bash[28559]: cluster 2023-11-17T07:27:21.791715+0000 mgr.smithi097.swvonl (mgr.14180) 1752 : cluster [DBG] pgmap v1231: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:24.092 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:24.092 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:24.950 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:25.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:24 smithi193 bash[28559]: cluster 2023-11-17T07:27:23.792353+0000 mgr.smithi097.swvonl (mgr.14180) 1753 : cluster [DBG] pgmap v1232: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:24 smithi097 bash[24958]: cluster 2023-11-17T07:27:23.792353+0000 mgr.smithi097.swvonl (mgr.14180) 1753 : cluster [DBG] pgmap v1232: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:25.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:26.149 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:25 smithi097 bash[24958]: audit 2023-11-17T07:27:24.086118+0000 mgr.smithi097.swvonl (mgr.14180) 1754 : audit [DBG] from='client.15376 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:26.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:25 smithi193 bash[28559]: audit 2023-11-17T07:27:24.086118+0000 mgr.smithi097.swvonl (mgr.14180) 1754 : audit [DBG] from='client.15376 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:27.155 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:26 smithi097 bash[24958]: cluster 2023-11-17T07:27:25.793647+0000 mgr.smithi097.swvonl (mgr.14180) 1755 : cluster [DBG] pgmap v1233: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:27.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:26 smithi193 bash[28559]: cluster 2023-11-17T07:27:25.793647+0000 mgr.smithi097.swvonl (mgr.14180) 1755 : cluster [DBG] pgmap v1233: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:28 smithi097 bash[24958]: cluster 2023-11-17T07:27:27.794395+0000 mgr.smithi097.swvonl (mgr.14180) 1756 : cluster [DBG] pgmap v1234: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:29.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:28 smithi193 bash[28559]: cluster 2023-11-17T07:27:27.794395+0000 mgr.smithi097.swvonl (mgr.14180) 1756 : cluster [DBG] pgmap v1234: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:29.463 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:29.463 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:30.365 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:31.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:30 smithi097 bash[24958]: audit 2023-11-17T07:27:29.451913+0000 mgr.smithi097.swvonl (mgr.14180) 1757 : audit [DBG] from='client.15380 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:31.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:30 smithi097 bash[24958]: cluster 2023-11-17T07:27:29.795300+0000 mgr.smithi097.swvonl (mgr.14180) 1758 : cluster [DBG] pgmap v1235: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:31.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:30 smithi193 bash[28559]: audit 2023-11-17T07:27:29.451913+0000 mgr.smithi097.swvonl (mgr.14180) 1757 : audit [DBG] from='client.15380 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:31.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:30 smithi193 bash[28559]: cluster 2023-11-17T07:27:29.795300+0000 mgr.smithi097.swvonl (mgr.14180) 1758 : cluster [DBG] pgmap v1235: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:31.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:32.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:32 smithi097 bash[24958]: cluster 2023-11-17T07:27:31.795889+0000 mgr.smithi097.swvonl (mgr.14180) 1759 : cluster [DBG] pgmap v1236: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:33.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:32 smithi193 bash[28559]: cluster 2023-11-17T07:27:31.795889+0000 mgr.smithi097.swvonl (mgr.14180) 1759 : cluster [DBG] pgmap v1236: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:35.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:34 smithi097 bash[24958]: cluster 2023-11-17T07:27:33.796925+0000 mgr.smithi097.swvonl (mgr.14180) 1760 : cluster [DBG] pgmap v1237: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:35.181 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:35.182 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:34 smithi193 bash[28559]: cluster 2023-11-17T07:27:33.796925+0000 mgr.smithi097.swvonl (mgr.14180) 1760 : cluster [DBG] pgmap v1237: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:36.053 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:37.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:37.256 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:36 smithi097 bash[24958]: audit 2023-11-17T07:27:35.171356+0000 mgr.smithi097.swvonl (mgr.14180) 1761 : audit [DBG] from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:37.256 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:36 smithi097 bash[24958]: cluster 2023-11-17T07:27:35.798346+0000 mgr.smithi097.swvonl (mgr.14180) 1762 : cluster [DBG] pgmap v1238: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:37.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:36 smithi193 bash[28559]: audit 2023-11-17T07:27:35.171356+0000 mgr.smithi097.swvonl (mgr.14180) 1761 : audit [DBG] from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:37.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:36 smithi193 bash[28559]: cluster 2023-11-17T07:27:35.798346+0000 mgr.smithi097.swvonl (mgr.14180) 1762 : cluster [DBG] pgmap v1238: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:39.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:38 smithi097 bash[24958]: cluster 2023-11-17T07:27:37.799059+0000 mgr.smithi097.swvonl (mgr.14180) 1763 : cluster [DBG] pgmap v1239: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:39.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:38 smithi193 bash[28559]: cluster 2023-11-17T07:27:37.799059+0000 mgr.smithi097.swvonl (mgr.14180) 1763 : cluster [DBG] pgmap v1239: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:40.790 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:40.790 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:41.093 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:40 smithi097 bash[24958]: cluster 2023-11-17T07:27:39.800376+0000 mgr.smithi097.swvonl (mgr.14180) 1764 : cluster [DBG] pgmap v1240: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:41.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:40 smithi193 bash[28559]: cluster 2023-11-17T07:27:39.800376+0000 mgr.smithi097.swvonl (mgr.14180) 1764 : cluster [DBG] pgmap v1240: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:41.715 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:42.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:41 smithi097 bash[24958]: audit 2023-11-17T07:27:40.779562+0000 mgr.smithi097.swvonl (mgr.14180) 1765 : audit [DBG] from='client.15388 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:42.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:41 smithi193 bash[28559]: audit 2023-11-17T07:27:40.779562+0000 mgr.smithi097.swvonl (mgr.14180) 1765 : audit [DBG] from='client.15388 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:42.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:43.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:42 smithi097 bash[24958]: cluster 2023-11-17T07:27:41.801560+0000 mgr.smithi097.swvonl (mgr.14180) 1766 : cluster [DBG] pgmap v1241: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:43.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:42 smithi193 bash[28559]: cluster 2023-11-17T07:27:41.801560+0000 mgr.smithi097.swvonl (mgr.14180) 1766 : cluster [DBG] pgmap v1241: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:45.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:44 smithi193 bash[28559]: cluster 2023-11-17T07:27:43.802794+0000 mgr.smithi097.swvonl (mgr.14180) 1767 : cluster [DBG] pgmap v1242: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:44 smithi097 bash[24958]: cluster 2023-11-17T07:27:43.802794+0000 mgr.smithi097.swvonl (mgr.14180) 1767 : cluster [DBG] pgmap v1242: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:46.549 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:46.549 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:47.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:46 smithi097 bash[24958]: cluster 2023-11-17T07:27:45.804131+0000 mgr.smithi097.swvonl (mgr.14180) 1768 : cluster [DBG] pgmap v1243: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:47.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:46 smithi193 bash[28559]: cluster 2023-11-17T07:27:45.804131+0000 mgr.smithi097.swvonl (mgr.14180) 1768 : cluster [DBG] pgmap v1243: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:47.529 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:48.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:47 smithi097 bash[24958]: audit 2023-11-17T07:27:46.538362+0000 mgr.smithi097.swvonl (mgr.14180) 1769 : audit [DBG] from='client.15392 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:48.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:47 smithi193 bash[28559]: audit 2023-11-17T07:27:46.538362+0000 mgr.smithi097.swvonl (mgr.14180) 1769 : audit [DBG] from='client.15392 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:48.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:49.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:48 smithi097 bash[24958]: cluster 2023-11-17T07:27:47.805102+0000 mgr.smithi097.swvonl (mgr.14180) 1770 : cluster [DBG] pgmap v1244: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:49.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:48 smithi193 bash[28559]: cluster 2023-11-17T07:27:47.805102+0000 mgr.smithi097.swvonl (mgr.14180) 1770 : cluster [DBG] pgmap v1244: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:51.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:50 smithi097 bash[24958]: cluster 2023-11-17T07:27:49.805801+0000 mgr.smithi097.swvonl (mgr.14180) 1771 : cluster [DBG] pgmap v1245: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:51.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:50 smithi193 bash[28559]: cluster 2023-11-17T07:27:49.805801+0000 mgr.smithi097.swvonl (mgr.14180) 1771 : cluster [DBG] pgmap v1245: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:27:52.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:52.242 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:53.077 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:52 smithi097 bash[24958]: cluster 2023-11-17T07:27:51.806489+0000 mgr.smithi097.swvonl (mgr.14180) 1772 : cluster [DBG] pgmap v1246: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:53.078 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:53.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:52 smithi193 bash[28559]: cluster 2023-11-17T07:27:51.806489+0000 mgr.smithi097.swvonl (mgr.14180) 1772 : cluster [DBG] pgmap v1246: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:54.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:27:54.092 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:53 smithi097 bash[24958]: audit 2023-11-17T07:27:52.235833+0000 mgr.smithi097.swvonl (mgr.14180) 1773 : audit [DBG] from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:54.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:53 smithi193 bash[28559]: audit 2023-11-17T07:27:52.235833+0000 mgr.smithi097.swvonl (mgr.14180) 1773 : audit [DBG] from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:55.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:54 smithi193 bash[28559]: cluster 2023-11-17T07:27:53.807529+0000 mgr.smithi097.swvonl (mgr.14180) 1774 : cluster [DBG] pgmap v1247: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:55.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:54 smithi097 bash[24958]: cluster 2023-11-17T07:27:53.807529+0000 mgr.smithi097.swvonl (mgr.14180) 1774 : cluster [DBG] pgmap v1247: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:57.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:56 smithi097 bash[24958]: cluster 2023-11-17T07:27:55.808666+0000 mgr.smithi097.swvonl (mgr.14180) 1775 : cluster [DBG] pgmap v1248: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:57.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:56 smithi193 bash[28559]: cluster 2023-11-17T07:27:55.808666+0000 mgr.smithi097.swvonl (mgr.14180) 1775 : cluster [DBG] pgmap v1248: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:57.732 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:27:57.732 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:27:58.625 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:27:59.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:58 smithi097 bash[24958]: audit 2023-11-17T07:27:57.720250+0000 mgr.smithi097.swvonl (mgr.14180) 1776 : audit [DBG] from='client.15400 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:59.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:27:58 smithi097 bash[24958]: cluster 2023-11-17T07:27:57.809469+0000 mgr.smithi097.swvonl (mgr.14180) 1777 : cluster [DBG] pgmap v1249: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:59.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:58 smithi193 bash[28559]: audit 2023-11-17T07:27:57.720250+0000 mgr.smithi097.swvonl (mgr.14180) 1776 : audit [DBG] from='client.15400 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:27:59.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:27:58 smithi193 bash[28559]: cluster 2023-11-17T07:27:57.809469+0000 mgr.smithi097.swvonl (mgr.14180) 1777 : cluster [DBG] pgmap v1249: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:27:59.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:01.114 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:00 smithi097 bash[24958]: cluster 2023-11-17T07:27:59.810659+0000 mgr.smithi097.swvonl (mgr.14180) 1778 : cluster [DBG] pgmap v1250: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:01.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:00 smithi193 bash[28559]: cluster 2023-11-17T07:27:59.810659+0000 mgr.smithi097.swvonl (mgr.14180) 1778 : cluster [DBG] pgmap v1250: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:03.285 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:03.285 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:03.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:02 smithi097 bash[24958]: cluster 2023-11-17T07:28:01.811832+0000 mgr.smithi097.swvonl (mgr.14180) 1779 : cluster [DBG] pgmap v1251: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:02 smithi193 bash[28559]: cluster 2023-11-17T07:28:01.811832+0000 mgr.smithi097.swvonl (mgr.14180) 1779 : cluster [DBG] pgmap v1251: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:04.146 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:05.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:05.157 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:04 smithi097 bash[24958]: audit 2023-11-17T07:28:03.272500+0000 mgr.smithi097.swvonl (mgr.14180) 1780 : audit [DBG] from='client.15404 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:05.158 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:04 smithi097 bash[24958]: cluster 2023-11-17T07:28:03.812960+0000 mgr.smithi097.swvonl (mgr.14180) 1781 : cluster [DBG] pgmap v1252: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:05.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:04 smithi193 bash[28559]: audit 2023-11-17T07:28:03.272500+0000 mgr.smithi097.swvonl (mgr.14180) 1780 : audit [DBG] from='client.15404 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:05.283 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:04 smithi193 bash[28559]: cluster 2023-11-17T07:28:03.812960+0000 mgr.smithi097.swvonl (mgr.14180) 1781 : cluster [DBG] pgmap v1252: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:07.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:06 smithi097 bash[24958]: cluster 2023-11-17T07:28:05.814331+0000 mgr.smithi097.swvonl (mgr.14180) 1782 : cluster [DBG] pgmap v1253: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:07.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:06 smithi193 bash[28559]: cluster 2023-11-17T07:28:05.814331+0000 mgr.smithi097.swvonl (mgr.14180) 1782 : cluster [DBG] pgmap v1253: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:09.025 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:09.025 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:09.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:08 smithi097 bash[24958]: cluster 2023-11-17T07:28:07.815313+0000 mgr.smithi097.swvonl (mgr.14180) 1783 : cluster [DBG] pgmap v1254: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:09.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:08 smithi193 bash[28559]: cluster 2023-11-17T07:28:07.815313+0000 mgr.smithi097.swvonl (mgr.14180) 1783 : cluster [DBG] pgmap v1254: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:09.932 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:10.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:09 smithi097 bash[24958]: audit 2023-11-17T07:28:09.014912+0000 mgr.smithi097.swvonl (mgr.14180) 1784 : audit [DBG] from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:10.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:09 smithi097 bash[24958]: audit 2023-11-17T07:28:09.788799+0000 mon.smithi097 (mon.0) 1115 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:28:10.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:09 smithi193 bash[28559]: audit 2023-11-17T07:28:09.014912+0000 mgr.smithi097.swvonl (mgr.14180) 1784 : audit [DBG] from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:10.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:09 smithi193 bash[28559]: audit 2023-11-17T07:28:09.788799+0000 mon.smithi097 (mon.0) 1115 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:28:10.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:11 smithi097 bash[24958]: cluster 2023-11-17T07:28:09.816485+0000 mgr.smithi097.swvonl (mgr.14180) 1785 : cluster [DBG] pgmap v1255: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:11 smithi097 bash[24958]: audit 2023-11-17T07:28:10.064118+0000 mon.smithi097 (mon.0) 1116 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:11 smithi193 bash[28559]: cluster 2023-11-17T07:28:09.816485+0000 mgr.smithi097.swvonl (mgr.14180) 1785 : cluster [DBG] pgmap v1255: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:11 smithi193 bash[28559]: audit 2023-11-17T07:28:10.064118+0000 mon.smithi097 (mon.0) 1116 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:13.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:13 smithi097 bash[24958]: cluster 2023-11-17T07:28:11.817573+0000 mgr.smithi097.swvonl (mgr.14180) 1786 : cluster [DBG] pgmap v1256: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:13.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:13 smithi193 bash[28559]: cluster 2023-11-17T07:28:11.817573+0000 mgr.smithi097.swvonl (mgr.14180) 1786 : cluster [DBG] pgmap v1256: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:14.518 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:14.519 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:15.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:15 smithi097 bash[24958]: cluster 2023-11-17T07:28:13.818795+0000 mgr.smithi097.swvonl (mgr.14180) 1787 : cluster [DBG] pgmap v1257: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:15.401 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:15.464 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:15 smithi193 bash[28559]: cluster 2023-11-17T07:28:13.818795+0000 mgr.smithi097.swvonl (mgr.14180) 1787 : cluster [DBG] pgmap v1257: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:16.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:16 smithi193 bash[28559]: audit 2023-11-17T07:28:14.508412+0000 mgr.smithi097.swvonl (mgr.14180) 1788 : audit [DBG] from='client.15412 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:16.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:16.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:16 smithi097 bash[24958]: audit 2023-11-17T07:28:14.508412+0000 mgr.smithi097.swvonl (mgr.14180) 1788 : audit [DBG] from='client.15412 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:17.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:17 smithi097 bash[24958]: cluster 2023-11-17T07:28:15.820237+0000 mgr.smithi097.swvonl (mgr.14180) 1789 : cluster [DBG] pgmap v1258: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:17.418 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:17 smithi193 bash[28559]: cluster 2023-11-17T07:28:15.820237+0000 mgr.smithi097.swvonl (mgr.14180) 1789 : cluster [DBG] pgmap v1258: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:19.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:19 smithi097 bash[24958]: cluster 2023-11-17T07:28:17.820828+0000 mgr.smithi097.swvonl (mgr.14180) 1790 : cluster [DBG] pgmap v1259: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:19.397 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:19 smithi193 bash[28559]: cluster 2023-11-17T07:28:17.820828+0000 mgr.smithi097.swvonl (mgr.14180) 1790 : cluster [DBG] pgmap v1259: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:20.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:20 smithi097 bash[24958]: cluster 2023-11-17T07:28:19.822131+0000 mgr.smithi097.swvonl (mgr.14180) 1791 : cluster [DBG] pgmap v1260: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:20.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:20 smithi193 bash[28559]: cluster 2023-11-17T07:28:19.822131+0000 mgr.smithi097.swvonl (mgr.14180) 1791 : cluster [DBG] pgmap v1260: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:21.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:21 smithi097 bash[24958]: audit 2023-11-17T07:28:20.123660+0000 mon.smithi097 (mon.0) 1117 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:21 smithi193 bash[28559]: audit 2023-11-17T07:28:20.123660+0000 mon.smithi097 (mon.0) 1117 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:22.344 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:22.344 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:23.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:22 smithi097 bash[24958]: cluster 2023-11-17T07:28:21.823153+0000 mgr.smithi097.swvonl (mgr.14180) 1792 : cluster [DBG] pgmap v1261: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:23.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:22 smithi193 bash[28559]: cluster 2023-11-17T07:28:21.823153+0000 mgr.smithi097.swvonl (mgr.14180) 1792 : cluster [DBG] pgmap v1261: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:23.350 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:24.271 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:23 smithi097 bash[24958]: audit 2023-11-17T07:28:22.332945+0000 mgr.smithi097.swvonl (mgr.14180) 1793 : audit [DBG] from='client.15416 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:23 smithi193 bash[28559]: audit 2023-11-17T07:28:22.332945+0000 mgr.smithi097.swvonl (mgr.14180) 1793 : audit [DBG] from='client.15416 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:24.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:25.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:24 smithi193 bash[28559]: cluster 2023-11-17T07:28:23.824130+0000 mgr.smithi097.swvonl (mgr.14180) 1794 : cluster [DBG] pgmap v1262: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:24 smithi097 bash[24958]: cluster 2023-11-17T07:28:23.824130+0000 mgr.smithi097.swvonl (mgr.14180) 1794 : cluster [DBG] pgmap v1262: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:27.041 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.638400+0000 mon.smithi097 (mon.0) 1118 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: cluster 2023-11-17T07:28:25.825426+0000 mgr.smithi097.swvonl (mgr.14180) 1795 : cluster [DBG] pgmap v1263: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.898722+0000 mon.smithi097 (mon.0) 1119 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.899645+0000 mon.smithi097 (mon.0) 1120 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.904901+0000 mon.smithi097 (mon.0) 1121 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: cluster 2023-11-17T07:28:25.905597+0000 mgr.smithi097.swvonl (mgr.14180) 1796 : cluster [DBG] pgmap v1264: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.910139+0000 mon.smithi097 (mon.0) 1122 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: audit 2023-11-17T07:28:25.924512+0000 mon.smithi097 (mon.0) 1123 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.100 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:26 smithi097 bash[24958]: cephadm 2023-11-17T07:28:25.929061+0000 mgr.smithi097.swvonl (mgr.14180) 1797 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.mvzrwq on smithi193 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.638400+0000 mon.smithi097 (mon.0) 1118 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: cluster 2023-11-17T07:28:25.825426+0000 mgr.smithi097.swvonl (mgr.14180) 1795 : cluster [DBG] pgmap v1263: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.898722+0000 mon.smithi097 (mon.0) 1119 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.899645+0000 mon.smithi097 (mon.0) 1120 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.904901+0000 mon.smithi097 (mon.0) 1121 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: cluster 2023-11-17T07:28:25.905597+0000 mgr.smithi097.swvonl (mgr.14180) 1796 : cluster [DBG] pgmap v1264: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.910139+0000 mon.smithi097 (mon.0) 1122 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: audit 2023-11-17T07:28:25.924512+0000 mon.smithi097 (mon.0) 1123 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:27.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:26 smithi193 bash[28559]: cephadm 2023-11-17T07:28:25.929061+0000 mgr.smithi097.swvonl (mgr.14180) 1797 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.mvzrwq on smithi193 2023-11-17T07:28:28.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:27 smithi097 bash[24958]: cluster 2023-11-17T07:28:26.635807+0000 mon.smithi097 (mon.0) 1124 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:28:28.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:27 smithi097 bash[24958]: cluster 2023-11-17T07:28:26.635883+0000 mon.smithi097 (mon.0) 1125 : cluster [INF] Cluster is now healthy 2023-11-17T07:28:28.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:27 smithi193 bash[28559]: cluster 2023-11-17T07:28:26.635807+0000 mon.smithi097 (mon.0) 1124 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:28:28.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:27 smithi193 bash[28559]: cluster 2023-11-17T07:28:26.635883+0000 mon.smithi097 (mon.0) 1125 : cluster [INF] Cluster is now healthy 2023-11-17T07:28:28.568 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:28.568 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:26:00.906076Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.cpeosp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:05.175705Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iclnsq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iclnsq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iclnsq\nDeploy daemon haproxy.nfs.foo.smithi193.iclnsq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.778824Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ujsimt on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ujsimt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ujsimt\nDeploy daemon haproxy.nfs.foo.smithi097.ujsimt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:27:09.781602Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.rkszfy on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:29.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:28 smithi097 bash[24958]: cluster 2023-11-17T07:28:27.906288+0000 mgr.smithi097.swvonl (mgr.14180) 1798 : cluster [DBG] pgmap v1265: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:29.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:28 smithi193 bash[28559]: cluster 2023-11-17T07:28:27.906288+0000 mgr.smithi097.swvonl (mgr.14180) 1798 : cluster [DBG] pgmap v1265: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:29.574 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:29.833 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:29 smithi097 bash[24958]: audit 2023-11-17T07:28:28.555591+0000 mgr.smithi097.swvonl (mgr.14180) 1799 : audit [DBG] from='client.15420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:30.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:29 smithi193 bash[28559]: audit 2023-11-17T07:28:28.555591+0000 mgr.smithi097.swvonl (mgr.14180) 1799 : audit [DBG] from='client.15420 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:30.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:31.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: cephadm 2023-11-17T07:28:29.718302+0000 mgr.smithi097.swvonl (mgr.14180) 1800 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:28:31.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: cephadm 2023-11-17T07:28:29.718915+0000 mgr.smithi097.swvonl (mgr.14180) 1801 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: audit 2023-11-17T07:28:29.719443+0000 mon.smithi097 (mon.0) 1126 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.mvzrwq"}]: dispatch 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: cephadm 2023-11-17T07:28:29.725168+0000 mgr.smithi097.swvonl (mgr.14180) 1802 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.mvzrwq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: cephadm 2023-11-17T07:28:29.728278+0000 mgr.smithi097.swvonl (mgr.14180) 1803 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lzgwuv on smithi097 2023-11-17T07:28:31.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:30 smithi097 bash[24958]: cluster 2023-11-17T07:28:29.907156+0000 mgr.smithi097.swvonl (mgr.14180) 1804 : cluster [DBG] pgmap v1266: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: cephadm 2023-11-17T07:28:29.718302+0000 mgr.smithi097.swvonl (mgr.14180) 1800 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:28:31.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: cephadm 2023-11-17T07:28:29.718915+0000 mgr.smithi097.swvonl (mgr.14180) 1801 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: audit 2023-11-17T07:28:29.719443+0000 mon.smithi097 (mon.0) 1126 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.mvzrwq"}]: dispatch 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: cephadm 2023-11-17T07:28:29.725168+0000 mgr.smithi097.swvonl (mgr.14180) 1802 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.mvzrwq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq 2023-11-17T07:28:31.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.mvzrwq ... 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: cephadm 2023-11-17T07:28:29.728278+0000 mgr.smithi097.swvonl (mgr.14180) 1803 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lzgwuv on smithi097 2023-11-17T07:28:31.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:30 smithi193 bash[28559]: cluster 2023-11-17T07:28:29.907156+0000 mgr.smithi097.swvonl (mgr.14180) 1804 : cluster [DBG] pgmap v1266: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:33.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:32 smithi097 bash[24958]: cluster 2023-11-17T07:28:31.907942+0000 mgr.smithi097.swvonl (mgr.14180) 1805 : cluster [DBG] pgmap v1267: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:33.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:32 smithi193 bash[28559]: cluster 2023-11-17T07:28:31.907942+0000 mgr.smithi097.swvonl (mgr.14180) 1805 : cluster [DBG] pgmap v1267: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:34.741 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:34.741 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:29.724915Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mvzrwq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\nDeploy daemon haproxy.nfs.foo.smithi193.mvzrwq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.024620Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lzgwuv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\nDeploy daemon haproxy.nfs.foo.smithi097.lzgwuv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.028713Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pojspz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:34.031455Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hbnxih on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cluster 2023-11-17T07:28:33.908824+0000 mgr.smithi097.swvonl (mgr.14180) 1806 : cluster [DBG] pgmap v1268: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cephadm 2023-11-17T07:28:34.022438+0000 mgr.smithi097.swvonl (mgr.14180) 1807 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cephadm 2023-11-17T07:28:34.022883+0000 mgr.smithi097.swvonl (mgr.14180) 1808 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: audit 2023-11-17T07:28:34.023477+0000 mon.smithi097 (mon.0) 1127 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lzgwuv"}]: dispatch 2023-11-17T07:28:35.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cephadm 2023-11-17T07:28:34.024830+0000 mgr.smithi097.swvonl (mgr.14180) 1809 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lzgwuv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cephadm 2023-11-17T07:28:34.028908+0000 mgr.smithi097.swvonl (mgr.14180) 1810 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pojspz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:28:35.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cephadm 2023-11-17T07:28:34.031558+0000 mgr.smithi097.swvonl (mgr.14180) 1811 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hbnxih on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:28:35.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: cluster 2023-11-17T07:28:34.032378+0000 mgr.smithi097.swvonl (mgr.14180) 1812 : cluster [DBG] pgmap v1269: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 249 B/s rd, 0 op/s 2023-11-17T07:28:35.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:34 smithi097 bash[24958]: audit 2023-11-17T07:28:34.733150+0000 mon.smithi097 (mon.0) 1128 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cluster 2023-11-17T07:28:33.908824+0000 mgr.smithi097.swvonl (mgr.14180) 1806 : cluster [DBG] pgmap v1268: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cephadm 2023-11-17T07:28:34.022438+0000 mgr.smithi097.swvonl (mgr.14180) 1807 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cephadm 2023-11-17T07:28:34.022883+0000 mgr.smithi097.swvonl (mgr.14180) 1808 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: audit 2023-11-17T07:28:34.023477+0000 mon.smithi097 (mon.0) 1127 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lzgwuv"}]: dispatch 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cephadm 2023-11-17T07:28:34.024830+0000 mgr.smithi097.swvonl (mgr.14180) 1809 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lzgwuv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgwuv ... 2023-11-17T07:28:35.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cephadm 2023-11-17T07:28:34.028908+0000 mgr.smithi097.swvonl (mgr.14180) 1810 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pojspz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cephadm 2023-11-17T07:28:34.031558+0000 mgr.smithi097.swvonl (mgr.14180) 1811 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hbnxih on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: cluster 2023-11-17T07:28:34.032378+0000 mgr.smithi097.swvonl (mgr.14180) 1812 : cluster [DBG] pgmap v1269: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 249 B/s rd, 0 op/s 2023-11-17T07:28:35.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:34 smithi193 bash[28559]: audit 2023-11-17T07:28:34.733150+0000 mon.smithi097 (mon.0) 1128 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:28:35.777 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:36.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:35 smithi097 bash[24958]: audit 2023-11-17T07:28:34.732007+0000 mgr.smithi097.swvonl (mgr.14180) 1813 : audit [DBG] from='client.15424 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:36.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:35 smithi097 bash[24958]: cluster 2023-11-17T07:28:34.962468+0000 mon.smithi097 (mon.0) 1129 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:28:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:35 smithi193 bash[28559]: audit 2023-11-17T07:28:34.732007+0000 mgr.smithi097.swvonl (mgr.14180) 1813 : audit [DBG] from='client.15424 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:36.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:35 smithi193 bash[28559]: cluster 2023-11-17T07:28:34.962468+0000 mon.smithi097 (mon.0) 1129 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:28:36.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:37.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:36 smithi097 bash[24958]: cluster 2023-11-17T07:28:36.033710+0000 mgr.smithi097.swvonl (mgr.14180) 1814 : cluster [DBG] pgmap v1270: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:37.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:36 smithi193 bash[28559]: cluster 2023-11-17T07:28:36.033710+0000 mgr.smithi097.swvonl (mgr.14180) 1814 : cluster [DBG] pgmap v1270: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:39.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:39 smithi097 bash[24958]: cluster 2023-11-17T07:28:38.034749+0000 mgr.smithi097.swvonl (mgr.14180) 1815 : cluster [DBG] pgmap v1271: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:39 smithi193 bash[28559]: cluster 2023-11-17T07:28:38.034749+0000 mgr.smithi097.swvonl (mgr.14180) 1815 : cluster [DBG] pgmap v1271: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:40.480 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:40.480 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:29.724915Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mvzrwq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\nDeploy daemon haproxy.nfs.foo.smithi193.mvzrwq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.024620Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lzgwuv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\nDeploy daemon haproxy.nfs.foo.smithi097.lzgwuv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.028713Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pojspz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:34.031455Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hbnxih on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:41.384 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:41 smithi097 bash[24958]: cluster 2023-11-17T07:28:40.036032+0000 mgr.smithi097.swvonl (mgr.14180) 1816 : cluster [DBG] pgmap v1272: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:41.385 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:41.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:41 smithi193 bash[28559]: cluster 2023-11-17T07:28:40.036032+0000 mgr.smithi097.swvonl (mgr.14180) 1816 : cluster [DBG] pgmap v1272: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:42.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:28:42.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:42 smithi097 bash[24958]: audit 2023-11-17T07:28:40.470366+0000 mgr.smithi097.swvonl (mgr.14180) 1817 : audit [DBG] from='client.15428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:42.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:42 smithi097 bash[24958]: cluster 2023-11-17T07:28:42.037039+0000 mgr.smithi097.swvonl (mgr.14180) 1818 : cluster [DBG] pgmap v1273: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:42.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:42 smithi193 bash[28559]: audit 2023-11-17T07:28:40.470366+0000 mgr.smithi097.swvonl (mgr.14180) 1817 : audit [DBG] from='client.15428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:42.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:42 smithi193 bash[28559]: cluster 2023-11-17T07:28:42.037039+0000 mgr.smithi097.swvonl (mgr.14180) 1818 : cluster [DBG] pgmap v1273: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:45 smithi097 bash[24958]: cluster 2023-11-17T07:28:44.037850+0000 mgr.smithi097.swvonl (mgr.14180) 1819 : cluster [DBG] pgmap v1274: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:56.930 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:28:56.930 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:19:02.647790Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:19:02.188597Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:19:02.648030Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:27:09.783971Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.txfbkc on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:29.724915Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.mvzrwq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-mvzrwq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.mvzrwq\nDeploy daemon haproxy.nfs.foo.smithi193.mvzrwq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.024620Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lzgwuv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgwuv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgwuv\nDeploy daemon haproxy.nfs.foo.smithi097.lzgwuv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:28:34.028713Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pojspz on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:28:34.031455Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hbnxih on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:19:02.189987Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:19:02.188826Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:19:02.188982Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:19:02.190194Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:19:02.189118Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:19:02.189249Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:19:02.648265Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:28:56.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:46 smithi097 bash[24958]: cluster 2023-11-17T07:28:46.039021+0000 mgr.smithi097.swvonl (mgr.14180) 1820 : cluster [DBG] pgmap v1275: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.931 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:45 smithi193 bash[28559]: cluster 2023-11-17T07:28:44.037850+0000 mgr.smithi097.swvonl (mgr.14180) 1819 : cluster [DBG] pgmap v1274: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:28:56.931 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:46 smithi193 bash[28559]: cluster 2023-11-17T07:28:46.039021+0000 mgr.smithi097.swvonl (mgr.14180) 1820 : cluster [DBG] pgmap v1275: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.931 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:47 smithi193 bash[28559]: audit 2023-11-17T07:28:46.483097+0000 mgr.smithi097.swvonl (mgr.14180) 1821 : audit [DBG] from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:56.932 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:48 smithi193 bash[28559]: cluster 2023-11-17T07:28:48.040018+0000 mgr.smithi097.swvonl (mgr.14180) 1822 : cluster [DBG] pgmap v1276: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.932 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:51 smithi193 bash[28559]: cluster 2023-11-17T07:28:50.041395+0000 mgr.smithi097.swvonl (mgr.14180) 1823 : cluster [DBG] pgmap v1277: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:56.932 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:52 smithi193 bash[28559]: cluster 2023-11-17T07:28:52.042528+0000 mgr.smithi097.swvonl (mgr.14180) 1824 : cluster [DBG] pgmap v1278: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.932 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:55 smithi193 bash[28559]: cluster 2023-11-17T07:28:54.043628+0000 mgr.smithi097.swvonl (mgr.14180) 1825 : cluster [DBG] pgmap v1279: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.932 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:56 smithi193 bash[28559]: cluster 2023-11-17T07:28:56.044994+0000 mgr.smithi097.swvonl (mgr.14180) 1826 : cluster [DBG] pgmap v1280: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.933 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:28:56.933 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:47 smithi097 bash[24958]: audit 2023-11-17T07:28:46.483097+0000 mgr.smithi097.swvonl (mgr.14180) 1821 : audit [DBG] from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:28:56.933 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:48 smithi097 bash[24958]: cluster 2023-11-17T07:28:48.040018+0000 mgr.smithi097.swvonl (mgr.14180) 1822 : cluster [DBG] pgmap v1276: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.933 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:51 smithi097 bash[24958]: cluster 2023-11-17T07:28:50.041395+0000 mgr.smithi097.swvonl (mgr.14180) 1823 : cluster [DBG] pgmap v1277: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:28:56.933 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:52 smithi097 bash[24958]: cluster 2023-11-17T07:28:52.042528+0000 mgr.smithi097.swvonl (mgr.14180) 1824 : cluster [DBG] pgmap v1278: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.933 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:55 smithi097 bash[24958]: cluster 2023-11-17T07:28:54.043628+0000 mgr.smithi097.swvonl (mgr.14180) 1825 : cluster [DBG] pgmap v1279: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:56.934 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:56 smithi097 bash[24958]: cluster 2023-11-17T07:28:56.044994+0000 mgr.smithi097.swvonl (mgr.14180) 1826 : cluster [DBG] pgmap v1280: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:28:57.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:30:59.780 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:28:58 smithi193 bash[28559]: cluster 2023-11-17T07:28:58.046109+0000 mgr.smithi097.swvonl (mgr.14180) 1827 : cluster [DBG] pgmap v1281: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:00 smithi193 bash[28559]: cluster 2023-11-17T07:29:00.047627+0000 mgr.smithi097.swvonl (mgr.14180) 1828 : cluster [DBG] pgmap v1282: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:02 smithi193 bash[28559]: cluster 2023-11-17T07:29:02.048750+0000 mgr.smithi097.swvonl (mgr.14180) 1829 : cluster [DBG] pgmap v1283: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:04 smithi193 bash[28559]: cluster 2023-11-17T07:29:04.049926+0000 mgr.smithi097.swvonl (mgr.14180) 1830 : cluster [DBG] pgmap v1284: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.781 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:06 smithi193 bash[28559]: cluster 2023-11-17T07:29:06.051408+0000 mgr.smithi097.swvonl (mgr.14180) 1831 : cluster [DBG] pgmap v1285: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:08 smithi193 bash[28559]: cluster 2023-11-17T07:29:08.052446+0000 mgr.smithi097.swvonl (mgr.14180) 1832 : cluster [DBG] pgmap v1286: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:10 smithi193 bash[28559]: cluster 2023-11-17T07:29:10.053834+0000 mgr.smithi097.swvonl (mgr.14180) 1833 : cluster [DBG] pgmap v1287: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:12 smithi193 bash[28559]: cluster 2023-11-17T07:29:12.054951+0000 mgr.smithi097.swvonl (mgr.14180) 1834 : cluster [DBG] pgmap v1288: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:14 smithi193 bash[28559]: cluster 2023-11-17T07:29:14.056051+0000 mgr.smithi097.swvonl (mgr.14180) 1835 : cluster [DBG] pgmap v1289: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:16 smithi193 bash[28559]: cluster 2023-11-17T07:29:16.057372+0000 mgr.smithi097.swvonl (mgr.14180) 1836 : cluster [DBG] pgmap v1290: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:18 smithi193 bash[28559]: cluster 2023-11-17T07:29:18.058237+0000 mgr.smithi097.swvonl (mgr.14180) 1837 : cluster [DBG] pgmap v1291: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:20 smithi193 bash[28559]: cluster 2023-11-17T07:29:20.059573+0000 mgr.smithi097.swvonl (mgr.14180) 1838 : cluster [DBG] pgmap v1292: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:22 smithi193 bash[28559]: cluster 2023-11-17T07:29:22.060887+0000 mgr.smithi097.swvonl (mgr.14180) 1839 : cluster [DBG] pgmap v1293: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.784 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:24 smithi193 bash[28559]: cluster 2023-11-17T07:29:24.061926+0000 mgr.smithi097.swvonl (mgr.14180) 1840 : cluster [DBG] pgmap v1294: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:26 smithi193 bash[28559]: cluster 2023-11-17T07:29:26.063243+0000 mgr.smithi097.swvonl (mgr.14180) 1841 : cluster [DBG] pgmap v1295: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:28 smithi193 bash[28559]: cluster 2023-11-17T07:29:28.064220+0000 mgr.smithi097.swvonl (mgr.14180) 1842 : cluster [DBG] pgmap v1296: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:30 smithi193 bash[28559]: cluster 2023-11-17T07:29:30.065647+0000 mgr.smithi097.swvonl (mgr.14180) 1843 : cluster [DBG] pgmap v1297: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:32 smithi193 bash[28559]: cluster 2023-11-17T07:29:32.066813+0000 mgr.smithi097.swvonl (mgr.14180) 1844 : cluster [DBG] pgmap v1298: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:34 smithi193 bash[28559]: audit 2023-11-17T07:29:34.035333+0000 mon.smithi097 (mon.0) 1130 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:35 smithi193 bash[28559]: cluster 2023-11-17T07:29:34.067594+0000 mgr.smithi097.swvonl (mgr.14180) 1845 : cluster [DBG] pgmap v1299: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:37 smithi193 bash[28559]: cluster 2023-11-17T07:29:36.068903+0000 mgr.smithi097.swvonl (mgr.14180) 1846 : cluster [DBG] pgmap v1300: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:39 smithi193 bash[28559]: cluster 2023-11-17T07:29:38.069866+0000 mgr.smithi097.swvonl (mgr.14180) 1847 : cluster [DBG] pgmap v1301: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:39.763698+0000 mon.smithi097 (mon.0) 1131 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: cluster 2023-11-17T07:29:40.071334+0000 mgr.smithi097.swvonl (mgr.14180) 1848 : cluster [DBG] pgmap v1302: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.360956+0000 mon.smithi097 (mon.0) 1132 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.657014+0000 mon.smithi097 (mon.0) 1133 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:30:59.785 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.658578+0000 mon.smithi097 (mon.0) 1134 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:30:59.786 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.667806+0000 mon.smithi097 (mon.0) 1135 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.673273+0000 mon.smithi097 (mon.0) 1136 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:40 smithi193 bash[28559]: audit 2023-11-17T07:29:40.689142+0000 mon.smithi097 (mon.0) 1137 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:41 smithi193 bash[28559]: cluster 2023-11-17T07:29:40.668678+0000 mgr.smithi097.swvonl (mgr.14180) 1849 : cluster [DBG] pgmap v1303: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:41 smithi193 bash[28559]: cephadm 2023-11-17T07:29:40.694364+0000 mgr.smithi097.swvonl (mgr.14180) 1850 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.zrrlzp on smithi193 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:41 smithi193 bash[28559]: cluster 2023-11-17T07:29:40.761022+0000 mon.smithi097 (mon.0) 1138 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:30:59.787 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:41 smithi193 bash[28559]: cluster 2023-11-17T07:29:40.761068+0000 mon.smithi097 (mon.0) 1139 : cluster [INF] Cluster is now healthy 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:41 smithi193 bash[28559]: audit 2023-11-17T07:29:41.564692+0000 mon.smithi097 (mon.0) 1140 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:43 smithi193 bash[28559]: cluster 2023-11-17T07:29:42.669377+0000 mgr.smithi097.swvonl (mgr.14180) 1851 : cluster [DBG] pgmap v1304: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:44 smithi193 bash[28559]: audit 2023-11-17T07:29:44.739255+0000 mon.smithi097 (mon.0) 1141 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.zrrlzp"}]: dispatch 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: cluster 2023-11-17T07:29:44.670175+0000 mgr.smithi097.swvonl (mgr.14180) 1852 : cluster [DBG] pgmap v1305: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: cephadm 2023-11-17T07:29:44.738004+0000 mgr.smithi097.swvonl (mgr.14180) 1853 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.788 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:30:59.789 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.790 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: cephadm 2023-11-17T07:29:44.738638+0000 mgr.smithi097.swvonl (mgr.14180) 1854 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: cephadm 2023-11-17T07:29:44.740747+0000 mgr.smithi097.swvonl (mgr.14180) 1855 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.zrrlzp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.791 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:45 smithi193 bash[28559]: cephadm 2023-11-17T07:29:44.745366+0000 mgr.smithi097.swvonl (mgr.14180) 1856 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.kraymr on smithi097 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:47 smithi193 bash[28559]: cluster 2023-11-17T07:29:46.671236+0000 mgr.smithi097.swvonl (mgr.14180) 1857 : cluster [DBG] pgmap v1306: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:49 smithi193 bash[28559]: cluster 2023-11-17T07:29:48.671968+0000 mgr.smithi097.swvonl (mgr.14180) 1858 : cluster [DBG] pgmap v1307: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:51 smithi193 bash[28559]: cluster 2023-11-17T07:29:50.673025+0000 mgr.smithi097.swvonl (mgr.14180) 1859 : cluster [DBG] pgmap v1308: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:53 smithi193 bash[28559]: cluster 2023-11-17T07:29:52.673752+0000 mgr.smithi097.swvonl (mgr.14180) 1860 : cluster [DBG] pgmap v1309: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:55 smithi193 bash[28559]: cluster 2023-11-17T07:29:54.674607+0000 mgr.smithi097.swvonl (mgr.14180) 1861 : cluster [DBG] pgmap v1310: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:57 smithi193 bash[28559]: cluster 2023-11-17T07:29:56.675598+0000 mgr.smithi097.swvonl (mgr.14180) 1862 : cluster [DBG] pgmap v1311: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:57 smithi193 bash[28559]: audit 2023-11-17T07:29:57.352601+0000 mon.smithi097 (mon.0) 1142 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.kraymr"}]: dispatch 2023-11-17T07:30:59.792 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cephadm 2023-11-17T07:29:57.351436+0000 mgr.smithi097.swvonl (mgr.14180) 1863 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.793 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.840 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.849 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.852 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.853 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cephadm 2023-11-17T07:29:57.351886+0000 mgr.smithi097.swvonl (mgr.14180) 1864 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cephadm 2023-11-17T07:29:57.354010+0000 mgr.smithi097.swvonl (mgr.14180) 1865 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.kraymr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.854 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cephadm 2023-11-17T07:29:57.356979+0000 mgr.smithi097.swvonl (mgr.14180) 1866 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vekayw on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cephadm 2023-11-17T07:29:57.359659+0000 mgr.smithi097.swvonl (mgr.14180) 1867 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cluster 2023-11-17T07:29:57.360907+0000 mgr.smithi097.swvonl (mgr.14180) 1868 : cluster [DBG] pgmap v1312: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:29:58 smithi193 bash[28559]: cluster 2023-11-17T07:29:57.915710+0000 mon.smithi097 (mon.0) 1143 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:00 smithi193 bash[28559]: cluster 2023-11-17T07:29:59.361967+0000 mgr.smithi097.swvonl (mgr.14180) 1869 : cluster [DBG] pgmap v1313: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:00 smithi193 bash[28559]: audit 2023-11-17T07:29:59.802538+0000 mon.smithi097 (mon.0) 1144 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:00 smithi193 bash[28559]: cluster 2023-11-17T07:30:00.000134+0000 mon.smithi097 (mon.0) 1145 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:02 smithi193 bash[28559]: cluster 2023-11-17T07:30:01.363314+0000 mgr.smithi097.swvonl (mgr.14180) 1870 : cluster [DBG] pgmap v1314: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.855 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:04 smithi193 bash[28559]: cluster 2023-11-17T07:30:03.364357+0000 mgr.smithi097.swvonl (mgr.14180) 1871 : cluster [DBG] pgmap v1315: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:06 smithi193 bash[28559]: cluster 2023-11-17T07:30:05.365738+0000 mgr.smithi097.swvonl (mgr.14180) 1872 : cluster [DBG] pgmap v1316: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:08 smithi193 bash[28559]: cluster 2023-11-17T07:30:07.366704+0000 mgr.smithi097.swvonl (mgr.14180) 1873 : cluster [DBG] pgmap v1317: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:10 smithi193 bash[28559]: cluster 2023-11-17T07:30:09.367823+0000 mgr.smithi097.swvonl (mgr.14180) 1874 : cluster [DBG] pgmap v1318: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:12 smithi193 bash[28559]: cluster 2023-11-17T07:30:11.369106+0000 mgr.smithi097.swvonl (mgr.14180) 1875 : cluster [DBG] pgmap v1319: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:12 smithi193 bash[28559]: audit 2023-11-17T07:30:11.565587+0000 mon.smithi097 (mon.0) 1146 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:14 smithi193 bash[28559]: cluster 2023-11-17T07:30:13.370144+0000 mgr.smithi097.swvonl (mgr.14180) 1876 : cluster [DBG] pgmap v1320: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:16 smithi193 bash[28559]: cluster 2023-11-17T07:30:15.371599+0000 mgr.smithi097.swvonl (mgr.14180) 1877 : cluster [DBG] pgmap v1321: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:18 smithi193 bash[28559]: cluster 2023-11-17T07:30:17.372566+0000 mgr.smithi097.swvonl (mgr.14180) 1878 : cluster [DBG] pgmap v1322: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:20 smithi193 bash[28559]: cluster 2023-11-17T07:30:19.373631+0000 mgr.smithi097.swvonl (mgr.14180) 1879 : cluster [DBG] pgmap v1323: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:22 smithi193 bash[28559]: cluster 2023-11-17T07:30:21.374950+0000 mgr.smithi097.swvonl (mgr.14180) 1880 : cluster [DBG] pgmap v1324: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:24 smithi193 bash[28559]: cluster 2023-11-17T07:30:23.376023+0000 mgr.smithi097.swvonl (mgr.14180) 1881 : cluster [DBG] pgmap v1325: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:26 smithi193 bash[28559]: cluster 2023-11-17T07:30:25.377325+0000 mgr.smithi097.swvonl (mgr.14180) 1882 : cluster [DBG] pgmap v1326: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:28 smithi193 bash[28559]: cluster 2023-11-17T07:30:27.378293+0000 mgr.smithi097.swvonl (mgr.14180) 1883 : cluster [DBG] pgmap v1327: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:30 smithi193 bash[28559]: cluster 2023-11-17T07:30:29.379405+0000 mgr.smithi097.swvonl (mgr.14180) 1884 : cluster [DBG] pgmap v1328: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.856 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:32 smithi193 bash[28559]: cluster 2023-11-17T07:30:31.380743+0000 mgr.smithi097.swvonl (mgr.14180) 1885 : cluster [DBG] pgmap v1329: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:34 smithi193 bash[28559]: cluster 2023-11-17T07:30:33.381758+0000 mgr.smithi097.swvonl (mgr.14180) 1886 : cluster [DBG] pgmap v1330: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:36 smithi193 bash[28559]: cluster 2023-11-17T07:30:35.383144+0000 mgr.smithi097.swvonl (mgr.14180) 1887 : cluster [DBG] pgmap v1331: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:38 smithi193 bash[28559]: cluster 2023-11-17T07:30:37.384111+0000 mgr.smithi097.swvonl (mgr.14180) 1888 : cluster [DBG] pgmap v1332: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:40 smithi193 bash[28559]: cluster 2023-11-17T07:30:39.385262+0000 mgr.smithi097.swvonl (mgr.14180) 1889 : cluster [DBG] pgmap v1333: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:42 smithi193 bash[28559]: cluster 2023-11-17T07:30:41.386724+0000 mgr.smithi097.swvonl (mgr.14180) 1890 : cluster [DBG] pgmap v1334: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:44 smithi193 bash[28559]: cluster 2023-11-17T07:30:43.387796+0000 mgr.smithi097.swvonl (mgr.14180) 1891 : cluster [DBG] pgmap v1335: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:46 smithi193 bash[28559]: cluster 2023-11-17T07:30:45.389198+0000 mgr.smithi097.swvonl (mgr.14180) 1892 : cluster [DBG] pgmap v1336: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:48 smithi193 bash[28559]: cluster 2023-11-17T07:30:47.390237+0000 mgr.smithi097.swvonl (mgr.14180) 1893 : cluster [DBG] pgmap v1337: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:50 smithi193 bash[28559]: cluster 2023-11-17T07:30:49.391501+0000 mgr.smithi097.swvonl (mgr.14180) 1894 : cluster [DBG] pgmap v1338: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:52 smithi193 bash[28559]: cluster 2023-11-17T07:30:51.392975+0000 mgr.smithi097.swvonl (mgr.14180) 1895 : cluster [DBG] pgmap v1339: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.857 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:54 smithi193 bash[28559]: cluster 2023-11-17T07:30:53.394055+0000 mgr.smithi097.swvonl (mgr.14180) 1896 : cluster [DBG] pgmap v1340: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:56 smithi193 bash[28559]: cluster 2023-11-17T07:30:55.395443+0000 mgr.smithi097.swvonl (mgr.14180) 1897 : cluster [DBG] pgmap v1341: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.364097+0000 mon.smithi097 (mon.0) 1147 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.643872+0000 mon.smithi097 (mon.0) 1148 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.937195+0000 mon.smithi097 (mon.0) 1149 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.938981+0000 mon.smithi097 (mon.0) 1150 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.947990+0000 mon.smithi097 (mon.0) 1151 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.955282+0000 mon.smithi097 (mon.0) 1152 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:58 smithi193 bash[28559]: audit 2023-11-17T07:30:57.970540+0000 mon.smithi097 (mon.0) 1153 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:59 smithi193 bash[28559]: cluster 2023-11-17T07:30:57.396432+0000 mgr.smithi097.swvonl (mgr.14180) 1898 : cluster [DBG] pgmap v1342: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:59 smithi193 bash[28559]: cluster 2023-11-17T07:30:57.948870+0000 mgr.smithi097.swvonl (mgr.14180) 1899 : cluster [DBG] pgmap v1343: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:59 smithi193 bash[28559]: cephadm 2023-11-17T07:30:57.978483+0000 mgr.smithi097.swvonl (mgr.14180) 1900 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.foeiun on smithi193 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:59 smithi193 bash[28559]: cluster 2023-11-17T07:30:57.990539+0000 mon.smithi097 (mon.0) 1154 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:30:59.858 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:30:59 smithi193 bash[28559]: cluster 2023-11-17T07:30:57.990577+0000 mon.smithi097 (mon.0) 1155 : cluster [INF] Cluster is now healthy 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:28:58 smithi097 bash[24958]: cluster 2023-11-17T07:28:58.046109+0000 mgr.smithi097.swvonl (mgr.14180) 1827 : cluster [DBG] pgmap v1281: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:00 smithi097 bash[24958]: cluster 2023-11-17T07:29:00.047627+0000 mgr.smithi097.swvonl (mgr.14180) 1828 : cluster [DBG] pgmap v1282: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:02 smithi097 bash[24958]: cluster 2023-11-17T07:29:02.048750+0000 mgr.smithi097.swvonl (mgr.14180) 1829 : cluster [DBG] pgmap v1283: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:04 smithi097 bash[24958]: cluster 2023-11-17T07:29:04.049926+0000 mgr.smithi097.swvonl (mgr.14180) 1830 : cluster [DBG] pgmap v1284: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:06 smithi097 bash[24958]: cluster 2023-11-17T07:29:06.051408+0000 mgr.smithi097.swvonl (mgr.14180) 1831 : cluster [DBG] pgmap v1285: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:08 smithi097 bash[24958]: cluster 2023-11-17T07:29:08.052446+0000 mgr.smithi097.swvonl (mgr.14180) 1832 : cluster [DBG] pgmap v1286: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:10 smithi097 bash[24958]: cluster 2023-11-17T07:29:10.053834+0000 mgr.smithi097.swvonl (mgr.14180) 1833 : cluster [DBG] pgmap v1287: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:12 smithi097 bash[24958]: cluster 2023-11-17T07:29:12.054951+0000 mgr.smithi097.swvonl (mgr.14180) 1834 : cluster [DBG] pgmap v1288: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:14 smithi097 bash[24958]: cluster 2023-11-17T07:29:14.056051+0000 mgr.smithi097.swvonl (mgr.14180) 1835 : cluster [DBG] pgmap v1289: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:16 smithi097 bash[24958]: cluster 2023-11-17T07:29:16.057372+0000 mgr.smithi097.swvonl (mgr.14180) 1836 : cluster [DBG] pgmap v1290: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:18 smithi097 bash[24958]: cluster 2023-11-17T07:29:18.058237+0000 mgr.smithi097.swvonl (mgr.14180) 1837 : cluster [DBG] pgmap v1291: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:20 smithi097 bash[24958]: cluster 2023-11-17T07:29:20.059573+0000 mgr.smithi097.swvonl (mgr.14180) 1838 : cluster [DBG] pgmap v1292: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.859 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:22 smithi097 bash[24958]: cluster 2023-11-17T07:29:22.060887+0000 mgr.smithi097.swvonl (mgr.14180) 1839 : cluster [DBG] pgmap v1293: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:24 smithi097 bash[24958]: cluster 2023-11-17T07:29:24.061926+0000 mgr.smithi097.swvonl (mgr.14180) 1840 : cluster [DBG] pgmap v1294: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:26 smithi097 bash[24958]: cluster 2023-11-17T07:29:26.063243+0000 mgr.smithi097.swvonl (mgr.14180) 1841 : cluster [DBG] pgmap v1295: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:28 smithi097 bash[24958]: cluster 2023-11-17T07:29:28.064220+0000 mgr.smithi097.swvonl (mgr.14180) 1842 : cluster [DBG] pgmap v1296: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.860 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:30 smithi097 bash[24958]: cluster 2023-11-17T07:29:30.065647+0000 mgr.smithi097.swvonl (mgr.14180) 1843 : cluster [DBG] pgmap v1297: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:32 smithi097 bash[24958]: cluster 2023-11-17T07:29:32.066813+0000 mgr.smithi097.swvonl (mgr.14180) 1844 : cluster [DBG] pgmap v1298: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:34 smithi097 bash[24958]: audit 2023-11-17T07:29:34.035333+0000 mon.smithi097 (mon.0) 1130 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:35 smithi097 bash[24958]: cluster 2023-11-17T07:29:34.067594+0000 mgr.smithi097.swvonl (mgr.14180) 1845 : cluster [DBG] pgmap v1299: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:37 smithi097 bash[24958]: cluster 2023-11-17T07:29:36.068903+0000 mgr.smithi097.swvonl (mgr.14180) 1846 : cluster [DBG] pgmap v1300: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:39 smithi097 bash[24958]: cluster 2023-11-17T07:29:38.069866+0000 mgr.smithi097.swvonl (mgr.14180) 1847 : cluster [DBG] pgmap v1301: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:39.763698+0000 mon.smithi097 (mon.0) 1131 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: cluster 2023-11-17T07:29:40.071334+0000 mgr.smithi097.swvonl (mgr.14180) 1848 : cluster [DBG] pgmap v1302: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.884 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.360956+0000 mon.smithi097 (mon.0) 1132 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.657014+0000 mon.smithi097 (mon.0) 1133 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.658578+0000 mon.smithi097 (mon.0) 1134 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.667806+0000 mon.smithi097 (mon.0) 1135 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.673273+0000 mon.smithi097 (mon.0) 1136 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:40 smithi097 bash[24958]: audit 2023-11-17T07:29:40.689142+0000 mon.smithi097 (mon.0) 1137 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:41 smithi097 bash[24958]: cluster 2023-11-17T07:29:40.668678+0000 mgr.smithi097.swvonl (mgr.14180) 1849 : cluster [DBG] pgmap v1303: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:41 smithi097 bash[24958]: cephadm 2023-11-17T07:29:40.694364+0000 mgr.smithi097.swvonl (mgr.14180) 1850 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.zrrlzp on smithi193 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:41 smithi097 bash[24958]: cluster 2023-11-17T07:29:40.761022+0000 mon.smithi097 (mon.0) 1138 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:41 smithi097 bash[24958]: cluster 2023-11-17T07:29:40.761068+0000 mon.smithi097 (mon.0) 1139 : cluster [INF] Cluster is now healthy 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:41 smithi097 bash[24958]: audit 2023-11-17T07:29:41.564692+0000 mon.smithi097 (mon.0) 1140 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:43 smithi097 bash[24958]: cluster 2023-11-17T07:29:42.669377+0000 mgr.smithi097.swvonl (mgr.14180) 1851 : cluster [DBG] pgmap v1304: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.885 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:44 smithi097 bash[24958]: audit 2023-11-17T07:29:44.739255+0000 mon.smithi097 (mon.0) 1141 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.zrrlzp"}]: dispatch 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: cluster 2023-11-17T07:29:44.670175+0000 mgr.smithi097.swvonl (mgr.14180) 1852 : cluster [DBG] pgmap v1305: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: cephadm 2023-11-17T07:29:44.738004+0000 mgr.smithi097.swvonl (mgr.14180) 1853 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.886 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.887 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: cephadm 2023-11-17T07:29:44.738638+0000 mgr.smithi097.swvonl (mgr.14180) 1854 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: cephadm 2023-11-17T07:29:44.740747+0000 mgr.smithi097.swvonl (mgr.14180) 1855 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.zrrlzp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.zrrlzp ... 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.888 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:45 smithi097 bash[24958]: cephadm 2023-11-17T07:29:44.745366+0000 mgr.smithi097.swvonl (mgr.14180) 1856 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.kraymr on smithi097 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:47 smithi097 bash[24958]: cluster 2023-11-17T07:29:46.671236+0000 mgr.smithi097.swvonl (mgr.14180) 1857 : cluster [DBG] pgmap v1306: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:49 smithi097 bash[24958]: cluster 2023-11-17T07:29:48.671968+0000 mgr.smithi097.swvonl (mgr.14180) 1858 : cluster [DBG] pgmap v1307: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:51 smithi097 bash[24958]: cluster 2023-11-17T07:29:50.673025+0000 mgr.smithi097.swvonl (mgr.14180) 1859 : cluster [DBG] pgmap v1308: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:53 smithi097 bash[24958]: cluster 2023-11-17T07:29:52.673752+0000 mgr.smithi097.swvonl (mgr.14180) 1860 : cluster [DBG] pgmap v1309: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:55 smithi097 bash[24958]: cluster 2023-11-17T07:29:54.674607+0000 mgr.smithi097.swvonl (mgr.14180) 1861 : cluster [DBG] pgmap v1310: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:57 smithi097 bash[24958]: cluster 2023-11-17T07:29:56.675598+0000 mgr.smithi097.swvonl (mgr.14180) 1862 : cluster [DBG] pgmap v1311: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:57 smithi097 bash[24958]: audit 2023-11-17T07:29:57.352601+0000 mon.smithi097 (mon.0) 1142 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.kraymr"}]: dispatch 2023-11-17T07:30:59.889 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cephadm 2023-11-17T07:29:57.351436+0000 mgr.smithi097.swvonl (mgr.14180) 1863 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:30:59.890 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.891 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cephadm 2023-11-17T07:29:57.351886+0000 mgr.smithi097.swvonl (mgr.14180) 1864 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cephadm 2023-11-17T07:29:57.354010+0000 mgr.smithi097.swvonl (mgr.14180) 1865 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.kraymr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.kraymr ... 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:30:59.892 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cephadm 2023-11-17T07:29:57.356979+0000 mgr.smithi097.swvonl (mgr.14180) 1866 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vekayw on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cephadm 2023-11-17T07:29:57.359659+0000 mgr.smithi097.swvonl (mgr.14180) 1867 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cluster 2023-11-17T07:29:57.360907+0000 mgr.smithi097.swvonl (mgr.14180) 1868 : cluster [DBG] pgmap v1312: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:29:58 smithi097 bash[24958]: cluster 2023-11-17T07:29:57.915710+0000 mon.smithi097 (mon.0) 1143 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:00 smithi097 bash[24958]: cluster 2023-11-17T07:29:59.361967+0000 mgr.smithi097.swvonl (mgr.14180) 1869 : cluster [DBG] pgmap v1313: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.893 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:00 smithi097 bash[24958]: audit 2023-11-17T07:29:59.802538+0000 mon.smithi097 (mon.0) 1144 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.929 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:00 smithi097 bash[24958]: cluster 2023-11-17T07:30:00.000134+0000 mon.smithi097 (mon.0) 1145 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:30:59.929 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:02 smithi097 bash[24958]: cluster 2023-11-17T07:30:01.363314+0000 mgr.smithi097.swvonl (mgr.14180) 1870 : cluster [DBG] pgmap v1314: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.929 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:04 smithi097 bash[24958]: cluster 2023-11-17T07:30:03.364357+0000 mgr.smithi097.swvonl (mgr.14180) 1871 : cluster [DBG] pgmap v1315: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.929 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:06 smithi097 bash[24958]: cluster 2023-11-17T07:30:05.365738+0000 mgr.smithi097.swvonl (mgr.14180) 1872 : cluster [DBG] pgmap v1316: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:08 smithi097 bash[24958]: cluster 2023-11-17T07:30:07.366704+0000 mgr.smithi097.swvonl (mgr.14180) 1873 : cluster [DBG] pgmap v1317: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 191 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:10 smithi097 bash[24958]: cluster 2023-11-17T07:30:09.367823+0000 mgr.smithi097.swvonl (mgr.14180) 1874 : cluster [DBG] pgmap v1318: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:12 smithi097 bash[24958]: cluster 2023-11-17T07:30:11.369106+0000 mgr.smithi097.swvonl (mgr.14180) 1875 : cluster [DBG] pgmap v1319: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:12 smithi097 bash[24958]: audit 2023-11-17T07:30:11.565587+0000 mon.smithi097 (mon.0) 1146 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:14 smithi097 bash[24958]: cluster 2023-11-17T07:30:13.370144+0000 mgr.smithi097.swvonl (mgr.14180) 1876 : cluster [DBG] pgmap v1320: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:16 smithi097 bash[24958]: cluster 2023-11-17T07:30:15.371599+0000 mgr.smithi097.swvonl (mgr.14180) 1877 : cluster [DBG] pgmap v1321: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:18 smithi097 bash[24958]: cluster 2023-11-17T07:30:17.372566+0000 mgr.smithi097.swvonl (mgr.14180) 1878 : cluster [DBG] pgmap v1322: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:20 smithi097 bash[24958]: cluster 2023-11-17T07:30:19.373631+0000 mgr.smithi097.swvonl (mgr.14180) 1879 : cluster [DBG] pgmap v1323: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:22 smithi097 bash[24958]: cluster 2023-11-17T07:30:21.374950+0000 mgr.smithi097.swvonl (mgr.14180) 1880 : cluster [DBG] pgmap v1324: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:24 smithi097 bash[24958]: cluster 2023-11-17T07:30:23.376023+0000 mgr.smithi097.swvonl (mgr.14180) 1881 : cluster [DBG] pgmap v1325: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:26 smithi097 bash[24958]: cluster 2023-11-17T07:30:25.377325+0000 mgr.smithi097.swvonl (mgr.14180) 1882 : cluster [DBG] pgmap v1326: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:28 smithi097 bash[24958]: cluster 2023-11-17T07:30:27.378293+0000 mgr.smithi097.swvonl (mgr.14180) 1883 : cluster [DBG] pgmap v1327: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:30 smithi097 bash[24958]: cluster 2023-11-17T07:30:29.379405+0000 mgr.smithi097.swvonl (mgr.14180) 1884 : cluster [DBG] pgmap v1328: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.930 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:32 smithi097 bash[24958]: cluster 2023-11-17T07:30:31.380743+0000 mgr.smithi097.swvonl (mgr.14180) 1885 : cluster [DBG] pgmap v1329: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:34 smithi097 bash[24958]: cluster 2023-11-17T07:30:33.381758+0000 mgr.smithi097.swvonl (mgr.14180) 1886 : cluster [DBG] pgmap v1330: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:36 smithi097 bash[24958]: cluster 2023-11-17T07:30:35.383144+0000 mgr.smithi097.swvonl (mgr.14180) 1887 : cluster [DBG] pgmap v1331: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:38 smithi097 bash[24958]: cluster 2023-11-17T07:30:37.384111+0000 mgr.smithi097.swvonl (mgr.14180) 1888 : cluster [DBG] pgmap v1332: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:40 smithi097 bash[24958]: cluster 2023-11-17T07:30:39.385262+0000 mgr.smithi097.swvonl (mgr.14180) 1889 : cluster [DBG] pgmap v1333: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:42 smithi097 bash[24958]: cluster 2023-11-17T07:30:41.386724+0000 mgr.smithi097.swvonl (mgr.14180) 1890 : cluster [DBG] pgmap v1334: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:44 smithi097 bash[24958]: cluster 2023-11-17T07:30:43.387796+0000 mgr.smithi097.swvonl (mgr.14180) 1891 : cluster [DBG] pgmap v1335: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:46 smithi097 bash[24958]: cluster 2023-11-17T07:30:45.389198+0000 mgr.smithi097.swvonl (mgr.14180) 1892 : cluster [DBG] pgmap v1336: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:48 smithi097 bash[24958]: cluster 2023-11-17T07:30:47.390237+0000 mgr.smithi097.swvonl (mgr.14180) 1893 : cluster [DBG] pgmap v1337: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:50 smithi097 bash[24958]: cluster 2023-11-17T07:30:49.391501+0000 mgr.smithi097.swvonl (mgr.14180) 1894 : cluster [DBG] pgmap v1338: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:52 smithi097 bash[24958]: cluster 2023-11-17T07:30:51.392975+0000 mgr.smithi097.swvonl (mgr.14180) 1895 : cluster [DBG] pgmap v1339: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:54 smithi097 bash[24958]: cluster 2023-11-17T07:30:53.394055+0000 mgr.smithi097.swvonl (mgr.14180) 1896 : cluster [DBG] pgmap v1340: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:56 smithi097 bash[24958]: cluster 2023-11-17T07:30:55.395443+0000 mgr.smithi097.swvonl (mgr.14180) 1897 : cluster [DBG] pgmap v1341: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.364097+0000 mon.smithi097 (mon.0) 1147 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:30:59.931 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.643872+0000 mon.smithi097 (mon.0) 1148 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.937195+0000 mon.smithi097 (mon.0) 1149 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.938981+0000 mon.smithi097 (mon.0) 1150 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.947990+0000 mon.smithi097 (mon.0) 1151 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.955282+0000 mon.smithi097 (mon.0) 1152 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:57 smithi097 bash[24958]: audit 2023-11-17T07:30:57.970540+0000 mon.smithi097 (mon.0) 1153 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:59 smithi097 bash[24958]: cluster 2023-11-17T07:30:57.396432+0000 mgr.smithi097.swvonl (mgr.14180) 1898 : cluster [DBG] pgmap v1342: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:59 smithi097 bash[24958]: cluster 2023-11-17T07:30:57.948870+0000 mgr.smithi097.swvonl (mgr.14180) 1899 : cluster [DBG] pgmap v1343: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:59 smithi097 bash[24958]: cephadm 2023-11-17T07:30:57.978483+0000 mgr.smithi097.swvonl (mgr.14180) 1900 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.foeiun on smithi193 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:59 smithi097 bash[24958]: cluster 2023-11-17T07:30:57.990539+0000 mon.smithi097 (mon.0) 1154 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:30:59.932 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:30:59 smithi097 bash[24958]: cluster 2023-11-17T07:30:57.990577+0000 mon.smithi097 (mon.0) 1155 : cluster [INF] Cluster is now healthy 2023-11-17T07:31:01.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:01 smithi097 bash[24958]: cluster 2023-11-17T07:30:59.950321+0000 mgr.smithi097.swvonl (mgr.14180) 1901 : cluster [DBG] pgmap v1344: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 387 B/s rd, 0 op/s 2023-11-17T07:31:01.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:01 smithi193 bash[28559]: cluster 2023-11-17T07:30:59.950321+0000 mgr.smithi097.swvonl (mgr.14180) 1901 : cluster [DBG] pgmap v1344: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 387 B/s rd, 0 op/s 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: cephadm 2023-11-17T07:31:01.922147+0000 mgr.smithi097.swvonl (mgr.14180) 1902 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:31:03.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: cephadm 2023-11-17T07:31:01.922713+0000 mgr.smithi097.swvonl (mgr.14180) 1903 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: audit 2023-11-17T07:31:01.923360+0000 mon.smithi097 (mon.0) 1156 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.foeiun"}]: dispatch 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: cephadm 2023-11-17T07:31:01.924960+0000 mgr.smithi097.swvonl (mgr.14180) 1904 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: cephadm 2023-11-17T07:31:01.928223+0000 mgr.smithi097.swvonl (mgr.14180) 1905 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aeoyhr on smithi097 2023-11-17T07:31:03.304 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:02 smithi097 bash[24958]: cluster 2023-11-17T07:31:01.950904+0000 mgr.smithi097.swvonl (mgr.14180) 1906 : cluster [DBG] pgmap v1345: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: cephadm 2023-11-17T07:31:01.922147+0000 mgr.smithi097.swvonl (mgr.14180) 1902 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: cephadm 2023-11-17T07:31:01.922713+0000 mgr.smithi097.swvonl (mgr.14180) 1903 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: audit 2023-11-17T07:31:01.923360+0000 mon.smithi097 (mon.0) 1156 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.foeiun"}]: dispatch 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: cephadm 2023-11-17T07:31:01.924960+0000 mgr.smithi097.swvonl (mgr.14180) 1904 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.foeiun ... 2023-11-17T07:31:03.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: cephadm 2023-11-17T07:31:01.928223+0000 mgr.smithi097.swvonl (mgr.14180) 1905 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aeoyhr on smithi097 2023-11-17T07:31:03.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:02 smithi193 bash[28559]: cluster 2023-11-17T07:31:01.950904+0000 mgr.smithi097.swvonl (mgr.14180) 1906 : cluster [DBG] pgmap v1345: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:31:04.048 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:04.048 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:44.740520Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.zrrlzp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-zrrlzp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.zrrlzp\nDeploy daemon haproxy.nfs.foo.smithi193.zrrlzp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:29:57.353840Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.kraymr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-kraymr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.kraymr\nDeploy daemon haproxy.nfs.foo.smithi097.kraymr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:29:57.356833Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vekayw on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:05.043 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:05.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:05 smithi193 bash[28559]: cluster 2023-11-17T07:31:03.951710+0000 mgr.smithi097.swvonl (mgr.14180) 1907 : cluster [DBG] pgmap v1346: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:31:05.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:05 smithi193 bash[28559]: audit 2023-11-17T07:31:04.037408+0000 mgr.smithi097.swvonl (mgr.14180) 1908 : audit [DBG] from='client.15436 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:05 smithi097 bash[24958]: cluster 2023-11-17T07:31:03.951710+0000 mgr.smithi097.swvonl (mgr.14180) 1907 : cluster [DBG] pgmap v1346: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:31:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:05 smithi097 bash[24958]: audit 2023-11-17T07:31:04.037408+0000 mgr.smithi097.swvonl (mgr.14180) 1908 : audit [DBG] from='client.15436 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:06.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:06.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:06 smithi097 bash[24958]: audit 2023-11-17T07:31:05.847336+0000 mon.smithi097 (mon.0) 1157 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aeoyhr"}]: dispatch 2023-11-17T07:31:06.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:06 smithi193 bash[28559]: audit 2023-11-17T07:31:05.847336+0000 mon.smithi097 (mon.0) 1157 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aeoyhr"}]: dispatch 2023-11-17T07:31:07.029 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cephadm 2023-11-17T07:31:05.846036+0000 mgr.smithi097.swvonl (mgr.14180) 1909 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.029 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.029 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.029 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:31:07.030 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.031 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cephadm 2023-11-17T07:31:05.846646+0000 mgr.smithi097.swvonl (mgr.14180) 1910 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cephadm 2023-11-17T07:31:05.848922+0000 mgr.smithi097.swvonl (mgr.14180) 1911 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cephadm 2023-11-17T07:31:05.846036+0000 mgr.smithi097.swvonl (mgr.14180) 1909 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:31:07.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cephadm 2023-11-17T07:31:05.846646+0000 mgr.smithi097.swvonl (mgr.14180) 1910 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cephadm 2023-11-17T07:31:05.848922+0000 mgr.smithi097.swvonl (mgr.14180) 1911 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cephadm 2023-11-17T07:31:05.853385+0000 mgr.smithi097.swvonl (mgr.14180) 1912 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:31:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cephadm 2023-11-17T07:31:05.855744+0000 mgr.smithi097.swvonl (mgr.14180) 1913 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:31:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cluster 2023-11-17T07:31:05.857123+0000 mgr.smithi097.swvonl (mgr.14180) 1914 : cluster [DBG] pgmap v1347: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:31:07.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:07 smithi097 bash[24958]: cluster 2023-11-17T07:31:06.008734+0000 mon.smithi097 (mon.0) 1158 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:31:07.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aeoyhr ... 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cephadm 2023-11-17T07:31:05.853385+0000 mgr.smithi097.swvonl (mgr.14180) 1912 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:31:07.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cephadm 2023-11-17T07:31:05.855744+0000 mgr.smithi097.swvonl (mgr.14180) 1913 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:31:07.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cluster 2023-11-17T07:31:05.857123+0000 mgr.smithi097.swvonl (mgr.14180) 1914 : cluster [DBG] pgmap v1347: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:31:07.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:07 smithi193 bash[28559]: cluster 2023-11-17T07:31:06.008734+0000 mon.smithi097 (mon.0) 1158 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:31:09.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:09 smithi097 bash[24958]: cluster 2023-11-17T07:31:07.858195+0000 mgr.smithi097.swvonl (mgr.14180) 1915 : cluster [DBG] pgmap v1348: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:31:09.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:09 smithi193 bash[28559]: cluster 2023-11-17T07:31:07.858195+0000 mgr.smithi097.swvonl (mgr.14180) 1915 : cluster [DBG] pgmap v1348: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:31:10.078 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:10.079 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:10.983 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:10 smithi097 bash[24958]: audit 2023-11-17T07:31:09.854854+0000 mon.smithi097 (mon.0) 1159 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:31:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:10 smithi097 bash[24958]: cluster 2023-11-17T07:31:09.858899+0000 mgr.smithi097.swvonl (mgr.14180) 1916 : cluster [DBG] pgmap v1349: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:31:11.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:10 smithi097 bash[24958]: audit 2023-11-17T07:31:10.072084+0000 mgr.smithi097.swvonl (mgr.14180) 1917 : audit [DBG] from='client.15440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:11.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:10 smithi193 bash[28559]: audit 2023-11-17T07:31:09.854854+0000 mon.smithi097 (mon.0) 1159 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:31:11.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:10 smithi193 bash[28559]: cluster 2023-11-17T07:31:09.858899+0000 mgr.smithi097.swvonl (mgr.14180) 1916 : cluster [DBG] pgmap v1349: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 343 B/s rd, 0 op/s 2023-11-17T07:31:11.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:10 smithi193 bash[28559]: audit 2023-11-17T07:31:10.072084+0000 mgr.smithi097.swvonl (mgr.14180) 1917 : audit [DBG] from='client.15440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:11.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:12 smithi097 bash[24958]: cluster 2023-11-17T07:31:11.859830+0000 mgr.smithi097.swvonl (mgr.14180) 1918 : cluster [DBG] pgmap v1350: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:13.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:12 smithi193 bash[28559]: cluster 2023-11-17T07:31:11.859830+0000 mgr.smithi097.swvonl (mgr.14180) 1918 : cluster [DBG] pgmap v1350: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:14 smithi193 bash[28559]: cluster 2023-11-17T07:31:13.860936+0000 mgr.smithi097.swvonl (mgr.14180) 1919 : cluster [DBG] pgmap v1351: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:15.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:14 smithi097 bash[24958]: cluster 2023-11-17T07:31:13.860936+0000 mgr.smithi097.swvonl (mgr.14180) 1919 : cluster [DBG] pgmap v1351: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:15.902 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:15.902 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:16.952 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:17.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:16 smithi097 bash[24958]: cluster 2023-11-17T07:31:15.862210+0000 mgr.smithi097.swvonl (mgr.14180) 1920 : cluster [DBG] pgmap v1352: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:17.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:16 smithi097 bash[24958]: audit 2023-11-17T07:31:15.890655+0000 mgr.smithi097.swvonl (mgr.14180) 1921 : audit [DBG] from='client.15444 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:17.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:16 smithi193 bash[28559]: cluster 2023-11-17T07:31:15.862210+0000 mgr.smithi097.swvonl (mgr.14180) 1920 : cluster [DBG] pgmap v1352: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:31:17.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:16 smithi193 bash[28559]: audit 2023-11-17T07:31:15.890655+0000 mgr.smithi097.swvonl (mgr.14180) 1921 : audit [DBG] from='client.15444 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:17.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:19.278 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:18 smithi097 bash[24958]: cluster 2023-11-17T07:31:17.863180+0000 mgr.smithi097.swvonl (mgr.14180) 1922 : cluster [DBG] pgmap v1353: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:19.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:18 smithi193 bash[28559]: cluster 2023-11-17T07:31:17.863180+0000 mgr.smithi097.swvonl (mgr.14180) 1922 : cluster [DBG] pgmap v1353: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:21.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:20 smithi097 bash[24958]: cluster 2023-11-17T07:31:19.864841+0000 mgr.smithi097.swvonl (mgr.14180) 1923 : cluster [DBG] pgmap v1354: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:21.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:20 smithi193 bash[28559]: cluster 2023-11-17T07:31:19.864841+0000 mgr.smithi097.swvonl (mgr.14180) 1923 : cluster [DBG] pgmap v1354: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:21.845 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:21.845 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:22.831 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:22 smithi097 bash[24958]: audit 2023-11-17T07:31:21.832858+0000 mgr.smithi097.swvonl (mgr.14180) 1924 : audit [DBG] from='client.15448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:22 smithi097 bash[24958]: cluster 2023-11-17T07:31:21.865973+0000 mgr.smithi097.swvonl (mgr.14180) 1925 : cluster [DBG] pgmap v1355: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:23.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:22 smithi193 bash[28559]: audit 2023-11-17T07:31:21.832858+0000 mgr.smithi097.swvonl (mgr.14180) 1924 : audit [DBG] from='client.15448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:23.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:22 smithi193 bash[28559]: cluster 2023-11-17T07:31:21.865973+0000 mgr.smithi097.swvonl (mgr.14180) 1925 : cluster [DBG] pgmap v1355: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:23.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:25.217 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:24 smithi097 bash[24958]: cluster 2023-11-17T07:31:23.866766+0000 mgr.smithi097.swvonl (mgr.14180) 1926 : cluster [DBG] pgmap v1356: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:25.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:24 smithi193 bash[28559]: cluster 2023-11-17T07:31:23.866766+0000 mgr.smithi097.swvonl (mgr.14180) 1926 : cluster [DBG] pgmap v1356: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:27.226 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:26 smithi097 bash[24958]: cluster 2023-11-17T07:31:25.868153+0000 mgr.smithi097.swvonl (mgr.14180) 1927 : cluster [DBG] pgmap v1357: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:27.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:26 smithi193 bash[28559]: cluster 2023-11-17T07:31:25.868153+0000 mgr.smithi097.swvonl (mgr.14180) 1927 : cluster [DBG] pgmap v1357: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:27.661 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:27.662 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:28.615 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:28 smithi097 bash[24958]: audit 2023-11-17T07:31:27.655308+0000 mgr.smithi097.swvonl (mgr.14180) 1928 : audit [DBG] from='client.15452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:29.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:28 smithi097 bash[24958]: cluster 2023-11-17T07:31:27.869156+0000 mgr.smithi097.swvonl (mgr.14180) 1929 : cluster [DBG] pgmap v1358: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:29.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:28 smithi193 bash[28559]: audit 2023-11-17T07:31:27.655308+0000 mgr.smithi097.swvonl (mgr.14180) 1928 : audit [DBG] from='client.15452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:29.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:28 smithi193 bash[28559]: cluster 2023-11-17T07:31:27.869156+0000 mgr.smithi097.swvonl (mgr.14180) 1929 : cluster [DBG] pgmap v1358: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:29.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:31.117 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:30 smithi097 bash[24958]: cluster 2023-11-17T07:31:29.870508+0000 mgr.smithi097.swvonl (mgr.14180) 1930 : cluster [DBG] pgmap v1359: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:31.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:30 smithi193 bash[28559]: cluster 2023-11-17T07:31:29.870508+0000 mgr.smithi097.swvonl (mgr.14180) 1930 : cluster [DBG] pgmap v1359: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:33.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:32 smithi097 bash[24958]: cluster 2023-11-17T07:31:31.871524+0000 mgr.smithi097.swvonl (mgr.14180) 1931 : cluster [DBG] pgmap v1360: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:33.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:32 smithi193 bash[28559]: cluster 2023-11-17T07:31:31.871524+0000 mgr.smithi097.swvonl (mgr.14180) 1931 : cluster [DBG] pgmap v1360: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:33.383 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:33.383 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:34.366 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:34 smithi193 bash[28559]: audit 2023-11-17T07:31:33.371967+0000 mgr.smithi097.swvonl (mgr.14180) 1932 : audit [DBG] from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:34 smithi193 bash[28559]: cluster 2023-11-17T07:31:33.872280+0000 mgr.smithi097.swvonl (mgr.14180) 1933 : cluster [DBG] pgmap v1361: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:34 smithi097 bash[24958]: audit 2023-11-17T07:31:33.371967+0000 mgr.smithi097.swvonl (mgr.14180) 1932 : audit [DBG] from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:34 smithi097 bash[24958]: cluster 2023-11-17T07:31:33.872280+0000 mgr.smithi097.swvonl (mgr.14180) 1933 : cluster [DBG] pgmap v1361: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:35.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:37.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:36 smithi097 bash[24958]: cluster 2023-11-17T07:31:35.873710+0000 mgr.smithi097.swvonl (mgr.14180) 1934 : cluster [DBG] pgmap v1362: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:37.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:36 smithi193 bash[28559]: cluster 2023-11-17T07:31:35.873710+0000 mgr.smithi097.swvonl (mgr.14180) 1934 : cluster [DBG] pgmap v1362: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:39.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:38 smithi097 bash[24958]: cluster 2023-11-17T07:31:37.874775+0000 mgr.smithi097.swvonl (mgr.14180) 1935 : cluster [DBG] pgmap v1363: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:39.213 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:39.213 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:39.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:38 smithi193 bash[28559]: cluster 2023-11-17T07:31:37.874775+0000 mgr.smithi097.swvonl (mgr.14180) 1935 : cluster [DBG] pgmap v1363: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:40.186 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:41.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:41.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:40 smithi097 bash[24958]: audit 2023-11-17T07:31:39.207019+0000 mgr.smithi097.swvonl (mgr.14180) 1936 : audit [DBG] from='client.15460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:41.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:40 smithi097 bash[24958]: cluster 2023-11-17T07:31:39.876282+0000 mgr.smithi097.swvonl (mgr.14180) 1937 : cluster [DBG] pgmap v1364: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:41.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:40 smithi193 bash[28559]: audit 2023-11-17T07:31:39.207019+0000 mgr.smithi097.swvonl (mgr.14180) 1936 : audit [DBG] from='client.15460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:41.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:40 smithi193 bash[28559]: cluster 2023-11-17T07:31:39.876282+0000 mgr.smithi097.swvonl (mgr.14180) 1937 : cluster [DBG] pgmap v1364: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:31:43.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:43 smithi097 bash[24958]: cluster 2023-11-17T07:31:41.877311+0000 mgr.smithi097.swvonl (mgr.14180) 1938 : cluster [DBG] pgmap v1365: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:43.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:43 smithi193 bash[28559]: cluster 2023-11-17T07:31:41.877311+0000 mgr.smithi097.swvonl (mgr.14180) 1938 : cluster [DBG] pgmap v1365: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:31:45.133 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:31:45.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:45 smithi193 bash[28559]: cluster 2023-11-17T07:31:43.878226+0000 mgr.smithi097.swvonl (mgr.14180) 1939 : cluster [DBG] pgmap v1366: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:45 smithi097 bash[24958]: cluster 2023-11-17T07:31:43.878226+0000 mgr.smithi097.swvonl (mgr.14180) 1939 : cluster [DBG] pgmap v1366: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:46.043 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:31:46.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:46 smithi097 bash[24958]: audit 2023-11-17T07:31:45.126414+0000 mgr.smithi097.swvonl (mgr.14180) 1940 : audit [DBG] from='client.15464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:46.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:46 smithi193 bash[28559]: audit 2023-11-17T07:31:45.126414+0000 mgr.smithi097.swvonl (mgr.14180) 1940 : audit [DBG] from='client.15464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:31:47.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:31:47.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:47 smithi097 bash[24958]: cluster 2023-11-17T07:31:45.879563+0000 mgr.smithi097.swvonl (mgr.14180) 1941 : cluster [DBG] pgmap v1367: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:47.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:47 smithi193 bash[28559]: cluster 2023-11-17T07:31:45.879563+0000 mgr.smithi097.swvonl (mgr.14180) 1941 : cluster [DBG] pgmap v1367: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:49.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:49 smithi097 bash[24958]: cluster 2023-11-17T07:31:47.880536+0000 mgr.smithi097.swvonl (mgr.14180) 1942 : cluster [DBG] pgmap v1368: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:49.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:49 smithi193 bash[28559]: cluster 2023-11-17T07:31:47.880536+0000 mgr.smithi097.swvonl (mgr.14180) 1942 : cluster [DBG] pgmap v1368: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:31:50.829 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:21.063 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:29:57.359519Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.dsbfcx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:01.924828Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.foeiun on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-foeiun\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.foeiun\nDeploy daemon haproxy.nfs.foo.smithi193.foeiun ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.848659Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aeoyhr on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aeoyhr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aeoyhr\nDeploy daemon haproxy.nfs.foo.smithi097.aeoyhr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:31:05.853189Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.hxndug on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:21.065 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:51 smithi097 bash[24958]: cluster 2023-11-17T07:31:49.881937+0000 mgr.smithi097.swvonl (mgr.14180) 1943 : cluster [DBG] pgmap v1369: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:21.067 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:21.078 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:51 smithi193 bash[28559]: cluster 2023-11-17T07:31:49.881937+0000 mgr.smithi097.swvonl (mgr.14180) 1943 : cluster [DBG] pgmap v1369: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:52 smithi193 bash[28559]: audit 2023-11-17T07:31:50.817661+0000 mgr.smithi097.swvonl (mgr.14180) 1944 : audit [DBG] from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:53 smithi193 bash[28559]: cluster 2023-11-17T07:31:51.883012+0000 mgr.smithi097.swvonl (mgr.14180) 1945 : cluster [DBG] pgmap v1370: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:55 smithi193 bash[28559]: cluster 2023-11-17T07:31:53.884085+0000 mgr.smithi097.swvonl (mgr.14180) 1946 : cluster [DBG] pgmap v1371: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:57 smithi193 bash[28559]: cluster 2023-11-17T07:31:55.885511+0000 mgr.smithi097.swvonl (mgr.14180) 1947 : cluster [DBG] pgmap v1372: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:31:59 smithi193 bash[28559]: cluster 2023-11-17T07:31:57.886488+0000 mgr.smithi097.swvonl (mgr.14180) 1948 : cluster [DBG] pgmap v1373: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.079 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:01 smithi193 bash[28559]: cluster 2023-11-17T07:31:59.887852+0000 mgr.smithi097.swvonl (mgr.14180) 1949 : cluster [DBG] pgmap v1374: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:03 smithi193 bash[28559]: cluster 2023-11-17T07:32:01.888878+0000 mgr.smithi097.swvonl (mgr.14180) 1950 : cluster [DBG] pgmap v1375: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:05 smithi193 bash[28559]: cluster 2023-11-17T07:32:03.889939+0000 mgr.smithi097.swvonl (mgr.14180) 1951 : cluster [DBG] pgmap v1376: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:06 smithi193 bash[28559]: audit 2023-11-17T07:32:05.859984+0000 mon.smithi097 (mon.0) 1160 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: cluster 2023-11-17T07:32:05.891271+0000 mgr.smithi097.swvonl (mgr.14180) 1952 : cluster [DBG] pgmap v1377: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: audit 2023-11-17T07:32:06.170926+0000 mon.smithi097 (mon.0) 1161 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: audit 2023-11-17T07:32:06.172293+0000 mon.smithi097 (mon.0) 1162 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: audit 2023-11-17T07:32:06.181420+0000 mon.smithi097 (mon.0) 1163 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: audit 2023-11-17T07:32:06.189469+0000 mon.smithi097 (mon.0) 1164 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:07 smithi193 bash[28559]: audit 2023-11-17T07:32:06.203812+0000 mon.smithi097 (mon.0) 1165 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:08 smithi193 bash[28559]: cluster 2023-11-17T07:32:06.182204+0000 mgr.smithi097.swvonl (mgr.14180) 1953 : cluster [DBG] pgmap v1378: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.080 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:08 smithi193 bash[28559]: cephadm 2023-11-17T07:32:06.207971+0000 mgr.smithi097.swvonl (mgr.14180) 1954 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.qglrhc on smithi193 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:08 smithi193 bash[28559]: cluster 2023-11-17T07:32:07.178193+0000 mon.smithi097 (mon.0) 1166 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:08 smithi193 bash[28559]: cluster 2023-11-17T07:32:07.178274+0000 mon.smithi097 (mon.0) 1167 : cluster [INF] Cluster is now healthy 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:10 smithi193 bash[28559]: cluster 2023-11-17T07:32:08.182994+0000 mgr.smithi097.swvonl (mgr.14180) 1955 : cluster [DBG] pgmap v1379: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:11 smithi193 bash[28559]: audit 2023-11-17T07:32:10.211185+0000 mon.smithi097 (mon.0) 1168 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.qglrhc"}]: dispatch 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: cluster 2023-11-17T07:32:10.184465+0000 mgr.smithi097.swvonl (mgr.14180) 1956 : cluster [DBG] pgmap v1380: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: cephadm 2023-11-17T07:32:10.210063+0000 mgr.smithi097.swvonl (mgr.14180) 1957 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.081 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.082 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.083 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.127 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.127 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.127 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: cephadm 2023-11-17T07:32:10.210533+0000 mgr.smithi097.swvonl (mgr.14180) 1958 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: cephadm 2023-11-17T07:32:10.212768+0000 mgr.smithi097.swvonl (mgr.14180) 1959 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.128 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: cephadm 2023-11-17T07:32:10.217725+0000 mgr.smithi097.swvonl (mgr.14180) 1960 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.gcllec on smithi097 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:12 smithi193 bash[28559]: audit 2023-11-17T07:32:11.568772+0000 mon.smithi097 (mon.0) 1169 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:14 smithi193 bash[28559]: cluster 2023-11-17T07:32:12.185146+0000 mgr.smithi097.swvonl (mgr.14180) 1961 : cluster [DBG] pgmap v1381: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:14 smithi193 bash[28559]: audit 2023-11-17T07:32:13.951617+0000 mon.smithi097 (mon.0) 1170 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.gcllec"}]: dispatch 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cephadm 2023-11-17T07:32:13.950685+0000 mgr.smithi097.swvonl (mgr.14180) 1962 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.129 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:32:21.130 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.131 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cephadm 2023-11-17T07:32:13.951145+0000 mgr.smithi097.swvonl (mgr.14180) 1963 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cephadm 2023-11-17T07:32:13.952974+0000 mgr.smithi097.swvonl (mgr.14180) 1964 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.132 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cephadm 2023-11-17T07:32:13.956774+0000 mgr.smithi097.swvonl (mgr.14180) 1965 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cephadm 2023-11-17T07:32:13.958584+0000 mgr.smithi097.swvonl (mgr.14180) 1966 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cluster 2023-11-17T07:32:13.959506+0000 mgr.smithi097.swvonl (mgr.14180) 1967 : cluster [DBG] pgmap v1382: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: cluster 2023-11-17T07:32:14.112172+0000 mon.smithi097 (mon.0) 1171 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:15 smithi193 bash[28559]: audit 2023-11-17T07:32:14.904899+0000 mon.smithi097 (mon.0) 1172 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:17 smithi193 bash[28559]: cluster 2023-11-17T07:32:15.960790+0000 mgr.smithi097.swvonl (mgr.14180) 1968 : cluster [DBG] pgmap v1383: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:32:21.133 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:19 smithi193 bash[28559]: cluster 2023-11-17T07:32:17.961744+0000 mgr.smithi097.swvonl (mgr.14180) 1969 : cluster [DBG] pgmap v1384: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:21.134 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:52 smithi097 bash[24958]: audit 2023-11-17T07:31:50.817661+0000 mgr.smithi097.swvonl (mgr.14180) 1944 : audit [DBG] from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:21.134 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:53 smithi097 bash[24958]: cluster 2023-11-17T07:31:51.883012+0000 mgr.smithi097.swvonl (mgr.14180) 1945 : cluster [DBG] pgmap v1370: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:55 smithi097 bash[24958]: cluster 2023-11-17T07:31:53.884085+0000 mgr.smithi097.swvonl (mgr.14180) 1946 : cluster [DBG] pgmap v1371: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:57 smithi097 bash[24958]: cluster 2023-11-17T07:31:55.885511+0000 mgr.smithi097.swvonl (mgr.14180) 1947 : cluster [DBG] pgmap v1372: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:31:59 smithi097 bash[24958]: cluster 2023-11-17T07:31:57.886488+0000 mgr.smithi097.swvonl (mgr.14180) 1948 : cluster [DBG] pgmap v1373: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:01 smithi097 bash[24958]: cluster 2023-11-17T07:31:59.887852+0000 mgr.smithi097.swvonl (mgr.14180) 1949 : cluster [DBG] pgmap v1374: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:03 smithi097 bash[24958]: cluster 2023-11-17T07:32:01.888878+0000 mgr.smithi097.swvonl (mgr.14180) 1950 : cluster [DBG] pgmap v1375: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:05 smithi097 bash[24958]: cluster 2023-11-17T07:32:03.889939+0000 mgr.smithi097.swvonl (mgr.14180) 1951 : cluster [DBG] pgmap v1376: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:06 smithi097 bash[24958]: audit 2023-11-17T07:32:05.859984+0000 mon.smithi097 (mon.0) 1160 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: cluster 2023-11-17T07:32:05.891271+0000 mgr.smithi097.swvonl (mgr.14180) 1952 : cluster [DBG] pgmap v1377: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:21.135 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: audit 2023-11-17T07:32:06.170926+0000 mon.smithi097 (mon.0) 1161 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: audit 2023-11-17T07:32:06.172293+0000 mon.smithi097 (mon.0) 1162 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: audit 2023-11-17T07:32:06.181420+0000 mon.smithi097 (mon.0) 1163 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: audit 2023-11-17T07:32:06.189469+0000 mon.smithi097 (mon.0) 1164 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:07 smithi097 bash[24958]: audit 2023-11-17T07:32:06.203812+0000 mon.smithi097 (mon.0) 1165 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:08 smithi097 bash[24958]: cluster 2023-11-17T07:32:06.182204+0000 mgr.smithi097.swvonl (mgr.14180) 1953 : cluster [DBG] pgmap v1378: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:08 smithi097 bash[24958]: cephadm 2023-11-17T07:32:06.207971+0000 mgr.smithi097.swvonl (mgr.14180) 1954 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.qglrhc on smithi193 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:08 smithi097 bash[24958]: cluster 2023-11-17T07:32:07.178193+0000 mon.smithi097 (mon.0) 1166 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:08 smithi097 bash[24958]: cluster 2023-11-17T07:32:07.178274+0000 mon.smithi097 (mon.0) 1167 : cluster [INF] Cluster is now healthy 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:10 smithi097 bash[24958]: cluster 2023-11-17T07:32:08.182994+0000 mgr.smithi097.swvonl (mgr.14180) 1955 : cluster [DBG] pgmap v1379: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:11 smithi097 bash[24958]: audit 2023-11-17T07:32:10.211185+0000 mon.smithi097 (mon.0) 1168 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.qglrhc"}]: dispatch 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: cluster 2023-11-17T07:32:10.184465+0000 mgr.smithi097.swvonl (mgr.14180) 1956 : cluster [DBG] pgmap v1380: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: cephadm 2023-11-17T07:32:10.210063+0000 mgr.smithi097.swvonl (mgr.14180) 1957 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.136 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.137 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.138 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: cephadm 2023-11-17T07:32:10.210533+0000 mgr.smithi097.swvonl (mgr.14180) 1958 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: cephadm 2023-11-17T07:32:10.212768+0000 mgr.smithi097.swvonl (mgr.14180) 1959 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc 2023-11-17T07:32:21.139 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.qglrhc ... 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.140 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: cephadm 2023-11-17T07:32:10.217725+0000 mgr.smithi097.swvonl (mgr.14180) 1960 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.gcllec on smithi097 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:12 smithi097 bash[24958]: audit 2023-11-17T07:32:11.568772+0000 mon.smithi097 (mon.0) 1169 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:14 smithi097 bash[24958]: cluster 2023-11-17T07:32:12.185146+0000 mgr.smithi097.swvonl (mgr.14180) 1961 : cluster [DBG] pgmap v1381: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:14 smithi097 bash[24958]: audit 2023-11-17T07:32:13.951617+0000 mon.smithi097 (mon.0) 1170 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.gcllec"}]: dispatch 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cephadm 2023-11-17T07:32:13.950685+0000 mgr.smithi097.swvonl (mgr.14180) 1962 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.141 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:32:21.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.143 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cephadm 2023-11-17T07:32:13.951145+0000 mgr.smithi097.swvonl (mgr.14180) 1963 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cephadm 2023-11-17T07:32:13.952974+0000 mgr.smithi097.swvonl (mgr.14180) 1964 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:32:21.144 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec 2023-11-17T07:32:21.174 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.gcllec ... 2023-11-17T07:32:21.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:32:21.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:32:21.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:32:21.175 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:32:21.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cephadm 2023-11-17T07:32:13.956774+0000 mgr.smithi097.swvonl (mgr.14180) 1965 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:32:21.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cephadm 2023-11-17T07:32:13.958584+0000 mgr.smithi097.swvonl (mgr.14180) 1966 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:32:21.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cluster 2023-11-17T07:32:13.959506+0000 mgr.smithi097.swvonl (mgr.14180) 1967 : cluster [DBG] pgmap v1382: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:32:21.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: cluster 2023-11-17T07:32:14.112172+0000 mon.smithi097 (mon.0) 1171 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:32:21.176 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:15 smithi097 bash[24958]: audit 2023-11-17T07:32:14.904899+0000 mon.smithi097 (mon.0) 1172 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:21.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:17 smithi097 bash[24958]: cluster 2023-11-17T07:32:15.960790+0000 mgr.smithi097.swvonl (mgr.14180) 1968 : cluster [DBG] pgmap v1383: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:32:21.177 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:19 smithi097 bash[24958]: cluster 2023-11-17T07:32:17.961744+0000 mgr.smithi097.swvonl (mgr.14180) 1969 : cluster [DBG] pgmap v1384: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:21.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:21 smithi097 bash[24958]: cluster 2023-11-17T07:32:19.963122+0000 mgr.smithi097.swvonl (mgr.14180) 1970 : cluster [DBG] pgmap v1385: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 347 B/s rd, 0 op/s 2023-11-17T07:32:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:21 smithi193 bash[28559]: cluster 2023-11-17T07:32:19.963122+0000 mgr.smithi097.swvonl (mgr.14180) 1970 : cluster [DBG] pgmap v1385: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 347 B/s rd, 0 op/s 2023-11-17T07:32:22.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:22 smithi097 bash[24958]: cluster 2023-11-17T07:32:21.964099+0000 mgr.smithi097.swvonl (mgr.14180) 1971 : cluster [DBG] pgmap v1386: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:23.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:22 smithi193 bash[28559]: cluster 2023-11-17T07:32:21.964099+0000 mgr.smithi097.swvonl (mgr.14180) 1971 : cluster [DBG] pgmap v1386: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:25.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:25 smithi193 bash[28559]: cluster 2023-11-17T07:32:23.965150+0000 mgr.smithi097.swvonl (mgr.14180) 1972 : cluster [DBG] pgmap v1387: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:25.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:25 smithi097 bash[24958]: cluster 2023-11-17T07:32:23.965150+0000 mgr.smithi097.swvonl (mgr.14180) 1972 : cluster [DBG] pgmap v1387: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:32:26.033 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:26.034 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:26.943 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:27.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:27 smithi097 bash[24958]: cluster 2023-11-17T07:32:25.966476+0000 mgr.smithi097.swvonl (mgr.14180) 1973 : cluster [DBG] pgmap v1388: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:27.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:27 smithi097 bash[24958]: audit 2023-11-17T07:32:26.026823+0000 mgr.smithi097.swvonl (mgr.14180) 1974 : audit [DBG] from='client.15472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:27.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:27 smithi097 bash[24958]: audit 2023-11-17T07:32:26.567477+0000 mon.smithi097 (mon.0) 1173 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:27.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:27 smithi193 bash[28559]: cluster 2023-11-17T07:32:25.966476+0000 mgr.smithi097.swvonl (mgr.14180) 1973 : cluster [DBG] pgmap v1388: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:27.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:27 smithi193 bash[28559]: audit 2023-11-17T07:32:26.026823+0000 mgr.smithi097.swvonl (mgr.14180) 1974 : audit [DBG] from='client.15472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:27.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:27 smithi193 bash[28559]: audit 2023-11-17T07:32:26.567477+0000 mon.smithi097 (mon.0) 1173 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:32:27.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:29.293 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:29 smithi097 bash[24958]: cluster 2023-11-17T07:32:27.967265+0000 mgr.smithi097.swvonl (mgr.14180) 1975 : cluster [DBG] pgmap v1389: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:29.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:29 smithi193 bash[28559]: cluster 2023-11-17T07:32:27.967265+0000 mgr.smithi097.swvonl (mgr.14180) 1975 : cluster [DBG] pgmap v1389: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:31.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:31 smithi097 bash[24958]: cluster 2023-11-17T07:32:29.968073+0000 mgr.smithi097.swvonl (mgr.14180) 1976 : cluster [DBG] pgmap v1390: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:31.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:31 smithi193 bash[28559]: cluster 2023-11-17T07:32:29.968073+0000 mgr.smithi097.swvonl (mgr.14180) 1976 : cluster [DBG] pgmap v1390: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:31.912 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:31.912 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:32.804 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:33.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:32 smithi097 bash[24958]: audit 2023-11-17T07:32:31.905180+0000 mgr.smithi097.swvonl (mgr.14180) 1977 : audit [DBG] from='client.15476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:33.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:32 smithi097 bash[24958]: cluster 2023-11-17T07:32:31.969138+0000 mgr.smithi097.swvonl (mgr.14180) 1978 : cluster [DBG] pgmap v1391: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:33.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:32 smithi193 bash[28559]: audit 2023-11-17T07:32:31.905180+0000 mgr.smithi097.swvonl (mgr.14180) 1977 : audit [DBG] from='client.15476 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:33.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:32 smithi193 bash[28559]: cluster 2023-11-17T07:32:31.969138+0000 mgr.smithi097.swvonl (mgr.14180) 1978 : cluster [DBG] pgmap v1391: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:33.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:35.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:35 smithi193 bash[28559]: cluster 2023-11-17T07:32:33.970266+0000 mgr.smithi097.swvonl (mgr.14180) 1979 : cluster [DBG] pgmap v1392: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:35.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:35 smithi097 bash[24958]: cluster 2023-11-17T07:32:33.970266+0000 mgr.smithi097.swvonl (mgr.14180) 1979 : cluster [DBG] pgmap v1392: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:37.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:37 smithi097 bash[24958]: cluster 2023-11-17T07:32:35.971582+0000 mgr.smithi097.swvonl (mgr.14180) 1980 : cluster [DBG] pgmap v1393: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:37.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:37 smithi193 bash[28559]: cluster 2023-11-17T07:32:35.971582+0000 mgr.smithi097.swvonl (mgr.14180) 1980 : cluster [DBG] pgmap v1393: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:37.574 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:37.575 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:38.539 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:39.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:39 smithi097 bash[24958]: audit 2023-11-17T07:32:37.564186+0000 mgr.smithi097.swvonl (mgr.14180) 1981 : audit [DBG] from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:39.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:39 smithi097 bash[24958]: cluster 2023-11-17T07:32:37.972797+0000 mgr.smithi097.swvonl (mgr.14180) 1982 : cluster [DBG] pgmap v1394: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:39.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:39 smithi193 bash[28559]: audit 2023-11-17T07:32:37.564186+0000 mgr.smithi097.swvonl (mgr.14180) 1981 : audit [DBG] from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:39.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:39 smithi193 bash[28559]: cluster 2023-11-17T07:32:37.972797+0000 mgr.smithi097.swvonl (mgr.14180) 1982 : cluster [DBG] pgmap v1394: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:39.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:41.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:41 smithi097 bash[24958]: cluster 2023-11-17T07:32:39.974281+0000 mgr.smithi097.swvonl (mgr.14180) 1983 : cluster [DBG] pgmap v1395: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:41.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:41 smithi193 bash[28559]: cluster 2023-11-17T07:32:39.974281+0000 mgr.smithi097.swvonl (mgr.14180) 1983 : cluster [DBG] pgmap v1395: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:43.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:42 smithi097 bash[24958]: cluster 2023-11-17T07:32:41.975261+0000 mgr.smithi097.swvonl (mgr.14180) 1984 : cluster [DBG] pgmap v1396: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:43.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:42 smithi193 bash[28559]: cluster 2023-11-17T07:32:41.975261+0000 mgr.smithi097.swvonl (mgr.14180) 1984 : cluster [DBG] pgmap v1396: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:43.508 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:43.509 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:44.479 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:45.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:45 smithi097 bash[24958]: audit 2023-11-17T07:32:43.494510+0000 mgr.smithi097.swvonl (mgr.14180) 1985 : audit [DBG] from='client.15484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:45.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:45 smithi097 bash[24958]: cluster 2023-11-17T07:32:43.976360+0000 mgr.smithi097.swvonl (mgr.14180) 1986 : cluster [DBG] pgmap v1397: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:45.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:45 smithi193 bash[28559]: audit 2023-11-17T07:32:43.494510+0000 mgr.smithi097.swvonl (mgr.14180) 1985 : audit [DBG] from='client.15484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:45.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:45 smithi193 bash[28559]: cluster 2023-11-17T07:32:43.976360+0000 mgr.smithi097.swvonl (mgr.14180) 1986 : cluster [DBG] pgmap v1397: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:45.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:47.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:47 smithi097 bash[24958]: cluster 2023-11-17T07:32:45.977670+0000 mgr.smithi097.swvonl (mgr.14180) 1987 : cluster [DBG] pgmap v1398: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:47.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:47 smithi193 bash[28559]: cluster 2023-11-17T07:32:45.977670+0000 mgr.smithi097.swvonl (mgr.14180) 1987 : cluster [DBG] pgmap v1398: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:49.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:49 smithi097 bash[24958]: cluster 2023-11-17T07:32:47.978419+0000 mgr.smithi097.swvonl (mgr.14180) 1988 : cluster [DBG] pgmap v1399: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:49.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:49 smithi193 bash[28559]: cluster 2023-11-17T07:32:47.978419+0000 mgr.smithi097.swvonl (mgr.14180) 1988 : cluster [DBG] pgmap v1399: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:49.373 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:49.373 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:50.295 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:51.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:51.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:51 smithi193 bash[28559]: audit 2023-11-17T07:32:49.361423+0000 mgr.smithi097.swvonl (mgr.14180) 1989 : audit [DBG] from='client.15488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:51.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:51 smithi193 bash[28559]: cluster 2023-11-17T07:32:49.979171+0000 mgr.smithi097.swvonl (mgr.14180) 1990 : cluster [DBG] pgmap v1400: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:51.309 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:51 smithi097 bash[24958]: audit 2023-11-17T07:32:49.361423+0000 mgr.smithi097.swvonl (mgr.14180) 1989 : audit [DBG] from='client.15488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:51.309 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:51 smithi097 bash[24958]: cluster 2023-11-17T07:32:49.979171+0000 mgr.smithi097.swvonl (mgr.14180) 1990 : cluster [DBG] pgmap v1400: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:32:53.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:52 smithi097 bash[24958]: cluster 2023-11-17T07:32:51.979887+0000 mgr.smithi097.swvonl (mgr.14180) 1991 : cluster [DBG] pgmap v1401: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:53.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:52 smithi193 bash[28559]: cluster 2023-11-17T07:32:51.979887+0000 mgr.smithi097.swvonl (mgr.14180) 1991 : cluster [DBG] pgmap v1401: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:55.082 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:32:55.082 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:32:55.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:55 smithi097 bash[24958]: cluster 2023-11-17T07:32:53.981037+0000 mgr.smithi097.swvonl (mgr.14180) 1992 : cluster [DBG] pgmap v1402: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:55.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:55 smithi193 bash[28559]: cluster 2023-11-17T07:32:53.981037+0000 mgr.smithi097.swvonl (mgr.14180) 1992 : cluster [DBG] pgmap v1402: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:56.105 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:32:56.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:56 smithi097 bash[24958]: audit 2023-11-17T07:32:55.071292+0000 mgr.smithi097.swvonl (mgr.14180) 1993 : audit [DBG] from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:56.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:56 smithi193 bash[28559]: audit 2023-11-17T07:32:55.071292+0000 mgr.smithi097.swvonl (mgr.14180) 1993 : audit [DBG] from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:32:57.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:32:57.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:57 smithi193 bash[28559]: cluster 2023-11-17T07:32:55.982474+0000 mgr.smithi097.swvonl (mgr.14180) 1994 : cluster [DBG] pgmap v1403: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:57.312 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:57 smithi097 bash[24958]: cluster 2023-11-17T07:32:55.982474+0000 mgr.smithi097.swvonl (mgr.14180) 1994 : cluster [DBG] pgmap v1403: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:59.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:32:59 smithi097 bash[24958]: cluster 2023-11-17T07:32:57.983104+0000 mgr.smithi097.swvonl (mgr.14180) 1995 : cluster [DBG] pgmap v1404: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:32:59.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:32:59 smithi193 bash[28559]: cluster 2023-11-17T07:32:57.983104+0000 mgr.smithi097.swvonl (mgr.14180) 1995 : cluster [DBG] pgmap v1404: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:01.156 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:01.157 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:01.450 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:01 smithi097 bash[24958]: cluster 2023-11-17T07:32:59.984474+0000 mgr.smithi097.swvonl (mgr.14180) 1996 : cluster [DBG] pgmap v1405: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:33:01.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:01 smithi193 bash[28559]: cluster 2023-11-17T07:32:59.984474+0000 mgr.smithi097.swvonl (mgr.14180) 1996 : cluster [DBG] pgmap v1405: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:33:02.072 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:02.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:02 smithi097 bash[24958]: audit 2023-11-17T07:33:01.143601+0000 mgr.smithi097.swvonl (mgr.14180) 1997 : audit [DBG] from='client.15496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:02 smithi193 bash[28559]: audit 2023-11-17T07:33:01.143601+0000 mgr.smithi097.swvonl (mgr.14180) 1997 : audit [DBG] from='client.15496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:03.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:03.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:03 smithi097 bash[24958]: cluster 2023-11-17T07:33:01.985614+0000 mgr.smithi097.swvonl (mgr.14180) 1998 : cluster [DBG] pgmap v1406: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:03.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:03 smithi193 bash[28559]: cluster 2023-11-17T07:33:01.985614+0000 mgr.smithi097.swvonl (mgr.14180) 1998 : cluster [DBG] pgmap v1406: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:05 smithi097 bash[24958]: cluster 2023-11-17T07:33:03.986769+0000 mgr.smithi097.swvonl (mgr.14180) 1999 : cluster [DBG] pgmap v1407: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:05.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:05 smithi193 bash[28559]: cluster 2023-11-17T07:33:03.986769+0000 mgr.smithi097.swvonl (mgr.14180) 1999 : cluster [DBG] pgmap v1407: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:07.036 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:07.036 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:07.341 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:07 smithi097 bash[24958]: cluster 2023-11-17T07:33:05.988278+0000 mgr.smithi097.swvonl (mgr.14180) 2000 : cluster [DBG] pgmap v1408: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:07.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:07 smithi193 bash[28559]: cluster 2023-11-17T07:33:05.988278+0000 mgr.smithi097.swvonl (mgr.14180) 2000 : cluster [DBG] pgmap v1408: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:08.068 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:08 smithi097 bash[24958]: audit 2023-11-17T07:33:07.024423+0000 mgr.smithi097.swvonl (mgr.14180) 2001 : audit [DBG] from='client.15500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:08.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:08 smithi193 bash[28559]: audit 2023-11-17T07:33:07.024423+0000 mgr.smithi097.swvonl (mgr.14180) 2001 : audit [DBG] from='client.15500 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:09.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:09.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:09 smithi097 bash[24958]: cluster 2023-11-17T07:33:07.989036+0000 mgr.smithi097.swvonl (mgr.14180) 2002 : cluster [DBG] pgmap v1409: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:09.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:09 smithi193 bash[28559]: cluster 2023-11-17T07:33:07.989036+0000 mgr.smithi097.swvonl (mgr.14180) 2002 : cluster [DBG] pgmap v1409: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:11.435 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:11 smithi097 bash[24958]: cluster 2023-11-17T07:33:09.990393+0000 mgr.smithi097.swvonl (mgr.14180) 2003 : cluster [DBG] pgmap v1410: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:33:11.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:11 smithi193 bash[28559]: cluster 2023-11-17T07:33:09.990393+0000 mgr.smithi097.swvonl (mgr.14180) 2003 : cluster [DBG] pgmap v1410: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:33:12.981 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:12.981 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:31:05.855647Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ukkons on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:13.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:12 smithi097 bash[24958]: cluster 2023-11-17T07:33:11.991432+0000 mgr.smithi097.swvonl (mgr.14180) 2004 : cluster [DBG] pgmap v1411: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:13.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:12 smithi193 bash[28559]: cluster 2023-11-17T07:33:11.991432+0000 mgr.smithi097.swvonl (mgr.14180) 2004 : cluster [DBG] pgmap v1411: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:14.005 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:14.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:13 smithi097 bash[24958]: audit 2023-11-17T07:33:12.968768+0000 mgr.smithi097.swvonl (mgr.14180) 2005 : audit [DBG] from='client.15504 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:14.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:13 smithi193 bash[28559]: audit 2023-11-17T07:33:12.968768+0000 mgr.smithi097.swvonl (mgr.14180) 2005 : audit [DBG] from='client.15504 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:15.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:15.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:13.962545+0000 mon.smithi097 (mon.0) 1174 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:33:15.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: cluster 2023-11-17T07:33:13.992622+0000 mgr.smithi097.swvonl (mgr.14180) 2006 : cluster [DBG] pgmap v1412: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:15.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:14.265015+0000 mon.smithi097 (mon.0) 1175 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:33:15.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:14.266398+0000 mon.smithi097 (mon.0) 1176 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:33:15.208 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:14.273498+0000 mon.smithi097 (mon.0) 1177 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:15.209 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:14.280066+0000 mon.smithi097 (mon.0) 1178 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:33:15.209 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:14 smithi097 bash[24958]: audit 2023-11-17T07:33:14.292858+0000 mon.smithi097 (mon.0) 1179 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:13.962545+0000 mon.smithi097 (mon.0) 1174 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: cluster 2023-11-17T07:33:13.992622+0000 mgr.smithi097.swvonl (mgr.14180) 2006 : cluster [DBG] pgmap v1412: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:14.265015+0000 mon.smithi097 (mon.0) 1175 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:14.266398+0000 mon.smithi097 (mon.0) 1176 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:14.273498+0000 mon.smithi097 (mon.0) 1177 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:14.280066+0000 mon.smithi097 (mon.0) 1178 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:33:15.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:14 smithi193 bash[28559]: audit 2023-11-17T07:33:14.292858+0000 mon.smithi097 (mon.0) 1179 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:16.291 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:15 smithi097 bash[24958]: cluster 2023-11-17T07:33:14.274332+0000 mgr.smithi097.swvonl (mgr.14180) 2007 : cluster [DBG] pgmap v1413: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:16.291 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:15 smithi097 bash[24958]: cephadm 2023-11-17T07:33:14.296679+0000 mgr.smithi097.swvonl (mgr.14180) 2008 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.uqjsca on smithi193 2023-11-17T07:33:16.291 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:15 smithi097 bash[24958]: cluster 2023-11-17T07:33:14.942226+0000 mon.smithi097 (mon.0) 1180 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:33:16.291 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:15 smithi097 bash[24958]: cluster 2023-11-17T07:33:14.942311+0000 mon.smithi097 (mon.0) 1181 : cluster [INF] Cluster is now healthy 2023-11-17T07:33:16.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:15 smithi193 bash[28559]: cluster 2023-11-17T07:33:14.274332+0000 mgr.smithi097.swvonl (mgr.14180) 2007 : cluster [DBG] pgmap v1413: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:16.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:15 smithi193 bash[28559]: cephadm 2023-11-17T07:33:14.296679+0000 mgr.smithi097.swvonl (mgr.14180) 2008 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.uqjsca on smithi193 2023-11-17T07:33:16.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:15 smithi193 bash[28559]: cluster 2023-11-17T07:33:14.942226+0000 mon.smithi097 (mon.0) 1180 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:33:16.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:15 smithi193 bash[28559]: cluster 2023-11-17T07:33:14.942311+0000 mon.smithi097 (mon.0) 1181 : cluster [INF] Cluster is now healthy 2023-11-17T07:33:18.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:17 smithi193 bash[28559]: cluster 2023-11-17T07:33:16.275407+0000 mgr.smithi097.swvonl (mgr.14180) 2009 : cluster [DBG] pgmap v1414: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:18.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:17 smithi097 bash[24958]: cluster 2023-11-17T07:33:16.275407+0000 mgr.smithi097.swvonl (mgr.14180) 2009 : cluster [DBG] pgmap v1414: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:18.785 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:18.785 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:32:10.212529Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.qglrhc on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-qglrhc\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.qglrhc\nDeploy daemon haproxy.nfs.foo.smithi193.qglrhc ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.952767Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.gcllec on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-gcllec\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.gcllec\nDeploy daemon haproxy.nfs.foo.smithi097.gcllec ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:32:13.956591Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vtcjzq on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:18.711302Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\nDeploy daemon haproxy.nfs.foo.smithi193.uqjsca ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:19.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:18 smithi097 bash[24958]: audit 2023-11-17T07:33:18.710031+0000 mon.smithi097 (mon.0) 1182 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.uqjsca"}]: dispatch 2023-11-17T07:33:19.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:18 smithi193 bash[28559]: audit 2023-11-17T07:33:18.710031+0000 mon.smithi097 (mon.0) 1182 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.uqjsca"}]: dispatch 2023-11-17T07:33:19.752 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: cluster 2023-11-17T07:33:18.276243+0000 mgr.smithi097.swvonl (mgr.14180) 2010 : cluster [DBG] pgmap v1415: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: cephadm 2023-11-17T07:33:18.709219+0000 mgr.smithi097.swvonl (mgr.14180) 2011 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.051 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: cephadm 2023-11-17T07:33:18.709473+0000 mgr.smithi097.swvonl (mgr.14180) 2012 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: cephadm 2023-11-17T07:33:18.711428+0000 mgr.smithi097.swvonl (mgr.14180) 2013 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.052 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: cephadm 2023-11-17T07:33:18.713999+0000 mgr.smithi097.swvonl (mgr.14180) 2014 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lgerth on smithi097 2023-11-17T07:33:20.053 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:19 smithi097 bash[24958]: audit 2023-11-17T07:33:18.778119+0000 mgr.smithi097.swvonl (mgr.14180) 2015 : audit [DBG] from='client.15508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: cluster 2023-11-17T07:33:18.276243+0000 mgr.smithi097.swvonl (mgr.14180) 2010 : cluster [DBG] pgmap v1415: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: cephadm 2023-11-17T07:33:18.709219+0000 mgr.smithi097.swvonl (mgr.14180) 2011 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:33:20.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: cephadm 2023-11-17T07:33:18.709473+0000 mgr.smithi097.swvonl (mgr.14180) 2012 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: cephadm 2023-11-17T07:33:18.711428+0000 mgr.smithi097.swvonl (mgr.14180) 2013 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.uqjsca ... 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: cephadm 2023-11-17T07:33:18.713999+0000 mgr.smithi097.swvonl (mgr.14180) 2014 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lgerth on smithi097 2023-11-17T07:33:20.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:19 smithi193 bash[28559]: audit 2023-11-17T07:33:18.778119+0000 mgr.smithi097.swvonl (mgr.14180) 2015 : audit [DBG] from='client.15508 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:20.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:22.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:21 smithi097 bash[24958]: cluster 2023-11-17T07:33:20.277418+0000 mgr.smithi097.swvonl (mgr.14180) 2016 : cluster [DBG] pgmap v1416: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:22.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:21 smithi193 bash[28559]: cluster 2023-11-17T07:33:20.277418+0000 mgr.smithi097.swvonl (mgr.14180) 2016 : cluster [DBG] pgmap v1416: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:23.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:22 smithi097 bash[24958]: audit 2023-11-17T07:33:22.843868+0000 mon.smithi097 (mon.0) 1183 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lgerth"}]: dispatch 2023-11-17T07:33:23.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:22 smithi193 bash[28559]: audit 2023-11-17T07:33:22.843868+0000 mon.smithi097 (mon.0) 1183 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lgerth"}]: dispatch 2023-11-17T07:33:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cluster 2023-11-17T07:33:22.278282+0000 mgr.smithi097.swvonl (mgr.14180) 2017 : cluster [DBG] pgmap v1417: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:24.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cephadm 2023-11-17T07:33:22.843165+0000 mgr.smithi097.swvonl (mgr.14180) 2018 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:33:24.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cephadm 2023-11-17T07:33:22.843438+0000 mgr.smithi097.swvonl (mgr.14180) 2019 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cephadm 2023-11-17T07:33:22.844858+0000 mgr.smithi097.swvonl (mgr.14180) 2020 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lgerth on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:33:24.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cephadm 2023-11-17T07:33:22.846688+0000 mgr.smithi097.swvonl (mgr.14180) 2021 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pdcvvd on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cephadm 2023-11-17T07:33:22.848169+0000 mgr.smithi097.swvonl (mgr.14180) 2022 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.rbshvz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cluster 2023-11-17T07:33:22.848835+0000 mgr.smithi097.swvonl (mgr.14180) 2023 : cluster [DBG] pgmap v1418: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2023-11-17T07:33:24.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:23 smithi097 bash[24958]: cluster 2023-11-17T07:33:22.980865+0000 mon.smithi097 (mon.0) 1184 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: cluster 2023-11-17T07:33:22.278282+0000 mgr.smithi097.swvonl (mgr.14180) 2017 : cluster [DBG] pgmap v1417: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: cephadm 2023-11-17T07:33:22.843165+0000 mgr.smithi097.swvonl (mgr.14180) 2018 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: cephadm 2023-11-17T07:33:22.843438+0000 mgr.smithi097.swvonl (mgr.14180) 2019 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: cephadm 2023-11-17T07:33:22.844858+0000 mgr.smithi097.swvonl (mgr.14180) 2020 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lgerth on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lgerth ... 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:33:24.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:33:24.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:33:24.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:23 smithi193 bash[28559]: cephadm 2023-11-17T07:33:22.846688+0000 mgr.smithi097.swvonl (mgr.14180) 2021 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.pdcvvd on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:33:24.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:24 smithi193 bash[28559]: cephadm 2023-11-17T07:33:22.848169+0000 mgr.smithi097.swvonl (mgr.14180) 2022 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.rbshvz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:33:24.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:24 smithi193 bash[28559]: cluster 2023-11-17T07:33:22.848835+0000 mgr.smithi097.swvonl (mgr.14180) 2023 : cluster [DBG] pgmap v1418: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2023-11-17T07:33:24.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:24 smithi193 bash[28559]: cluster 2023-11-17T07:33:22.980865+0000 mon.smithi097 (mon.0) 1184 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:33:25.053 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:25.053 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:18.711302Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\nDeploy daemon haproxy.nfs.foo.smithi193.uqjsca ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.844734Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lgerth on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\nDeploy daemon haproxy.nfs.foo.smithi097.lgerth ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.846596Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pdcvvd on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:22.848079Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbshvz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:25.992 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:26.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:25 smithi097 bash[24958]: cluster 2023-11-17T07:33:24.849696+0000 mgr.smithi097.swvonl (mgr.14180) 2024 : cluster [DBG] pgmap v1419: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:26.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:25 smithi097 bash[24958]: audit 2023-11-17T07:33:24.956056+0000 mon.smithi097 (mon.0) 1185 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:26.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:25 smithi097 bash[24958]: audit 2023-11-17T07:33:25.046707+0000 mgr.smithi097.swvonl (mgr.14180) 2025 : audit [DBG] from='client.15512 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:26.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:25 smithi193 bash[28559]: cluster 2023-11-17T07:33:24.849696+0000 mgr.smithi097.swvonl (mgr.14180) 2024 : cluster [DBG] pgmap v1419: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:26.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:25 smithi193 bash[28559]: audit 2023-11-17T07:33:24.956056+0000 mon.smithi097 (mon.0) 1185 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:33:26.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:25 smithi193 bash[28559]: audit 2023-11-17T07:33:25.046707+0000 mgr.smithi097.swvonl (mgr.14180) 2025 : audit [DBG] from='client.15512 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:26.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:28.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:27 smithi097 bash[24958]: cluster 2023-11-17T07:33:26.850761+0000 mgr.smithi097.swvonl (mgr.14180) 2026 : cluster [DBG] pgmap v1420: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:28.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:27 smithi193 bash[28559]: cluster 2023-11-17T07:33:26.850761+0000 mgr.smithi097.swvonl (mgr.14180) 2026 : cluster [DBG] pgmap v1420: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:30.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:29 smithi097 bash[24958]: cluster 2023-11-17T07:33:28.851720+0000 mgr.smithi097.swvonl (mgr.14180) 2027 : cluster [DBG] pgmap v1421: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:30.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:29 smithi193 bash[28559]: cluster 2023-11-17T07:33:28.851720+0000 mgr.smithi097.swvonl (mgr.14180) 2027 : cluster [DBG] pgmap v1421: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:30.823 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:33:30.823 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:18.711302Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\nDeploy daemon haproxy.nfs.foo.smithi193.uqjsca ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.844734Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lgerth on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\nDeploy daemon haproxy.nfs.foo.smithi097.lgerth ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.846596Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pdcvvd on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:22.848079Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbshvz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:33:31.859 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:33:32.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:31 smithi097 bash[24958]: audit 2023-11-17T07:33:30.812280+0000 mgr.smithi097.swvonl (mgr.14180) 2028 : audit [DBG] from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:32.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:31 smithi097 bash[24958]: cluster 2023-11-17T07:33:30.852901+0000 mgr.smithi097.swvonl (mgr.14180) 2029 : cluster [DBG] pgmap v1422: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:32.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:31 smithi193 bash[28559]: audit 2023-11-17T07:33:30.812280+0000 mgr.smithi097.swvonl (mgr.14180) 2028 : audit [DBG] from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:33:32.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:31 smithi193 bash[28559]: cluster 2023-11-17T07:33:30.852901+0000 mgr.smithi097.swvonl (mgr.14180) 2029 : cluster [DBG] pgmap v1422: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:32.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:33:34.235 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:33 smithi097 bash[24958]: cluster 2023-11-17T07:33:32.853878+0000 mgr.smithi097.swvonl (mgr.14180) 2030 : cluster [DBG] pgmap v1423: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:34.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:33 smithi193 bash[28559]: cluster 2023-11-17T07:33:32.853878+0000 mgr.smithi097.swvonl (mgr.14180) 2030 : cluster [DBG] pgmap v1423: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 193 B/s rd, 0 op/s 2023-11-17T07:33:36.267 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:35 smithi097 bash[24958]: cluster 2023-11-17T07:33:34.854701+0000 mgr.smithi097.swvonl (mgr.14180) 2031 : cluster [DBG] pgmap v1424: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:41.957 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:35:41.968 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:32:13.958485Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiedkd on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:18.711302Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.uqjsca on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-uqjsca\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.uqjsca\nDeploy daemon haproxy.nfs.foo.smithi193.uqjsca ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.844734Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lgerth on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lgerth\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lgerth\nDeploy daemon haproxy.nfs.foo.smithi097.lgerth ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:33:22.846596Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.pdcvvd on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:33:22.848079Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.rbshvz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:35:41.969 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:35:41.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:35 smithi193 bash[28559]: cluster 2023-11-17T07:33:34.854701+0000 mgr.smithi097.swvonl (mgr.14180) 2031 : cluster [DBG] pgmap v1424: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.178 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:37 smithi193 bash[28559]: audit 2023-11-17T07:33:36.697621+0000 mgr.smithi097.swvonl (mgr.14180) 2032 : audit [DBG] from='client.15520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:44.179 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:37 smithi193 bash[28559]: cluster 2023-11-17T07:33:36.855690+0000 mgr.smithi097.swvonl (mgr.14180) 2033 : cluster [DBG] pgmap v1425: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.179 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:39 smithi193 bash[28559]: cluster 2023-11-17T07:33:38.856932+0000 mgr.smithi097.swvonl (mgr.14180) 2034 : cluster [DBG] pgmap v1426: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.179 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:42 smithi193 bash[28559]: cluster 2023-11-17T07:33:40.858382+0000 mgr.smithi097.swvonl (mgr.14180) 2035 : cluster [DBG] pgmap v1427: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:44 smithi193 bash[28559]: cluster 2023-11-17T07:33:42.859466+0000 mgr.smithi097.swvonl (mgr.14180) 2036 : cluster [DBG] pgmap v1428: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:46 smithi193 bash[28559]: cluster 2023-11-17T07:33:44.860966+0000 mgr.smithi097.swvonl (mgr.14180) 2037 : cluster [DBG] pgmap v1429: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:48 smithi193 bash[28559]: cluster 2023-11-17T07:33:46.862233+0000 mgr.smithi097.swvonl (mgr.14180) 2038 : cluster [DBG] pgmap v1430: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:50 smithi193 bash[28559]: cluster 2023-11-17T07:33:48.863434+0000 mgr.smithi097.swvonl (mgr.14180) 2039 : cluster [DBG] pgmap v1431: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:52 smithi193 bash[28559]: cluster 2023-11-17T07:33:50.864773+0000 mgr.smithi097.swvonl (mgr.14180) 2040 : cluster [DBG] pgmap v1432: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:54 smithi193 bash[28559]: cluster 2023-11-17T07:33:52.865910+0000 mgr.smithi097.swvonl (mgr.14180) 2041 : cluster [DBG] pgmap v1433: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:56 smithi193 bash[28559]: cluster 2023-11-17T07:33:54.867290+0000 mgr.smithi097.swvonl (mgr.14180) 2042 : cluster [DBG] pgmap v1434: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:33:58 smithi193 bash[28559]: cluster 2023-11-17T07:33:56.868368+0000 mgr.smithi097.swvonl (mgr.14180) 2043 : cluster [DBG] pgmap v1435: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:00 smithi193 bash[28559]: cluster 2023-11-17T07:33:58.869534+0000 mgr.smithi097.swvonl (mgr.14180) 2044 : cluster [DBG] pgmap v1436: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:02 smithi193 bash[28559]: cluster 2023-11-17T07:34:00.870830+0000 mgr.smithi097.swvonl (mgr.14180) 2045 : cluster [DBG] pgmap v1437: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:04 smithi193 bash[28559]: cluster 2023-11-17T07:34:02.871777+0000 mgr.smithi097.swvonl (mgr.14180) 2046 : cluster [DBG] pgmap v1438: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:06 smithi193 bash[28559]: cluster 2023-11-17T07:34:04.873176+0000 mgr.smithi097.swvonl (mgr.14180) 2047 : cluster [DBG] pgmap v1439: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:08 smithi193 bash[28559]: cluster 2023-11-17T07:34:06.874283+0000 mgr.smithi097.swvonl (mgr.14180) 2048 : cluster [DBG] pgmap v1440: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.180 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:10 smithi193 bash[28559]: cluster 2023-11-17T07:34:08.875292+0000 mgr.smithi097.swvonl (mgr.14180) 2049 : cluster [DBG] pgmap v1441: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.181 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:12 smithi193 bash[28559]: cluster 2023-11-17T07:34:10.876638+0000 mgr.smithi097.swvonl (mgr.14180) 2050 : cluster [DBG] pgmap v1442: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:14 smithi193 bash[28559]: cluster 2023-11-17T07:34:12.877747+0000 mgr.smithi097.swvonl (mgr.14180) 2051 : cluster [DBG] pgmap v1443: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:16 smithi193 bash[28559]: cluster 2023-11-17T07:34:14.879176+0000 mgr.smithi097.swvonl (mgr.14180) 2052 : cluster [DBG] pgmap v1444: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:18 smithi193 bash[28559]: cluster 2023-11-17T07:34:16.880189+0000 mgr.smithi097.swvonl (mgr.14180) 2053 : cluster [DBG] pgmap v1445: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:20 smithi193 bash[28559]: cluster 2023-11-17T07:34:18.881402+0000 mgr.smithi097.swvonl (mgr.14180) 2054 : cluster [DBG] pgmap v1446: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:22 smithi193 bash[28559]: cluster 2023-11-17T07:34:20.882919+0000 mgr.smithi097.swvonl (mgr.14180) 2055 : cluster [DBG] pgmap v1447: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:23 smithi193 bash[28559]: audit 2023-11-17T07:34:22.852234+0000 mon.smithi097 (mon.0) 1186 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: cluster 2023-11-17T07:34:22.883696+0000 mgr.smithi097.swvonl (mgr.14180) 2056 : cluster [DBG] pgmap v1448: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: audit 2023-11-17T07:34:23.160843+0000 mon.smithi097 (mon.0) 1187 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: audit 2023-11-17T07:34:23.162082+0000 mon.smithi097 (mon.0) 1188 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: audit 2023-11-17T07:34:23.169488+0000 mon.smithi097 (mon.0) 1189 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.182 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: audit 2023-11-17T07:34:23.178043+0000 mon.smithi097 (mon.0) 1190 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:24 smithi193 bash[28559]: audit 2023-11-17T07:34:23.191499+0000 mon.smithi097 (mon.0) 1191 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:25 smithi193 bash[28559]: cluster 2023-11-17T07:34:23.170322+0000 mgr.smithi097.swvonl (mgr.14180) 2057 : cluster [DBG] pgmap v1449: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:25 smithi193 bash[28559]: cephadm 2023-11-17T07:34:23.194756+0000 mgr.smithi097.swvonl (mgr.14180) 2058 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.tskdgs on smithi193 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:25 smithi193 bash[28559]: cluster 2023-11-17T07:34:24.166305+0000 mon.smithi097 (mon.0) 1192 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:25 smithi193 bash[28559]: cluster 2023-11-17T07:34:24.166356+0000 mon.smithi097 (mon.0) 1193 : cluster [INF] Cluster is now healthy 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:27 smithi193 bash[28559]: cluster 2023-11-17T07:34:25.171336+0000 mgr.smithi097.swvonl (mgr.14180) 2059 : cluster [DBG] pgmap v1450: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:27 smithi193 bash[28559]: audit 2023-11-17T07:34:26.571662+0000 mon.smithi097 (mon.0) 1194 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:28 smithi193 bash[28559]: audit 2023-11-17T07:34:27.283400+0000 mon.smithi097 (mon.0) 1195 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.tskdgs"}]: dispatch 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: cluster 2023-11-17T07:34:27.172258+0000 mgr.smithi097.swvonl (mgr.14180) 2060 : cluster [DBG] pgmap v1451: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: cephadm 2023-11-17T07:34:27.282287+0000 mgr.smithi097.swvonl (mgr.14180) 2061 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.183 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.184 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:35:44.185 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.212 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.217 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.217 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.217 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.217 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.217 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.223 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: cephadm 2023-11-17T07:34:27.282792+0000 mgr.smithi097.swvonl (mgr.14180) 2062 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: cephadm 2023-11-17T07:34:27.284995+0000 mgr.smithi097.swvonl (mgr.14180) 2063 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.tskdgs on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.224 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:29 smithi193 bash[28559]: cephadm 2023-11-17T07:34:27.288866+0000 mgr.smithi097.swvonl (mgr.14180) 2064 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lzgrra on smithi097 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:31 smithi193 bash[28559]: cluster 2023-11-17T07:34:29.173008+0000 mgr.smithi097.swvonl (mgr.14180) 2065 : cluster [DBG] pgmap v1452: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:32 smithi193 bash[28559]: audit 2023-11-17T07:34:31.468628+0000 mon.smithi097 (mon.0) 1196 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lzgrra"}]: dispatch 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cluster 2023-11-17T07:34:31.173844+0000 mgr.smithi097.swvonl (mgr.14180) 2066 : cluster [DBG] pgmap v1453: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cephadm 2023-11-17T07:34:31.467337+0000 mgr.smithi097.swvonl (mgr.14180) 2067 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.225 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.226 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.227 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cephadm 2023-11-17T07:34:31.467848+0000 mgr.smithi097.swvonl (mgr.14180) 2068 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cephadm 2023-11-17T07:34:31.470202+0000 mgr.smithi097.swvonl (mgr.14180) 2069 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lzgrra on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.228 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.229 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.276 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.276 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.276 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.277 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.277 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.277 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cephadm 2023-11-17T07:34:31.473033+0000 mgr.smithi097.swvonl (mgr.14180) 2070 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.oulsqb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.277 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cephadm 2023-11-17T07:34:31.475312+0000 mgr.smithi097.swvonl (mgr.14180) 2071 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.277 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cluster 2023-11-17T07:34:31.476465+0000 mgr.smithi097.swvonl (mgr.14180) 2072 : cluster [DBG] pgmap v1454: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 238 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:33 smithi193 bash[28559]: cluster 2023-11-17T07:34:32.154298+0000 mon.smithi097 (mon.0) 1197 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:34 smithi193 bash[28559]: cluster 2023-11-17T07:34:33.477508+0000 mgr.smithi097.swvonl (mgr.14180) 2073 : cluster [DBG] pgmap v1455: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:36 smithi193 bash[28559]: audit 2023-11-17T07:34:35.009348+0000 mon.smithi097 (mon.0) 1198 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:37 smithi193 bash[28559]: cluster 2023-11-17T07:34:35.478799+0000 mgr.smithi097.swvonl (mgr.14180) 2074 : cluster [DBG] pgmap v1456: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:39 smithi193 bash[28559]: cluster 2023-11-17T07:34:37.479782+0000 mgr.smithi097.swvonl (mgr.14180) 2075 : cluster [DBG] pgmap v1457: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:41 smithi193 bash[28559]: cluster 2023-11-17T07:34:39.480888+0000 mgr.smithi097.swvonl (mgr.14180) 2076 : cluster [DBG] pgmap v1458: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:42 smithi193 bash[28559]: cluster 2023-11-17T07:34:41.482236+0000 mgr.smithi097.swvonl (mgr.14180) 2077 : cluster [DBG] pgmap v1459: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:42 smithi193 bash[28559]: audit 2023-11-17T07:34:41.571664+0000 mon.smithi097 (mon.0) 1199 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.285 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:44 smithi193 bash[28559]: cluster 2023-11-17T07:34:43.483292+0000 mgr.smithi097.swvonl (mgr.14180) 2078 : cluster [DBG] pgmap v1460: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:46 smithi193 bash[28559]: cluster 2023-11-17T07:34:45.484759+0000 mgr.smithi097.swvonl (mgr.14180) 2079 : cluster [DBG] pgmap v1461: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:48 smithi193 bash[28559]: cluster 2023-11-17T07:34:47.485702+0000 mgr.smithi097.swvonl (mgr.14180) 2080 : cluster [DBG] pgmap v1462: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:50 smithi193 bash[28559]: cluster 2023-11-17T07:34:49.486804+0000 mgr.smithi097.swvonl (mgr.14180) 2081 : cluster [DBG] pgmap v1463: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:52 smithi193 bash[28559]: cluster 2023-11-17T07:34:51.488261+0000 mgr.smithi097.swvonl (mgr.14180) 2082 : cluster [DBG] pgmap v1464: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:54 smithi193 bash[28559]: cluster 2023-11-17T07:34:53.489473+0000 mgr.smithi097.swvonl (mgr.14180) 2083 : cluster [DBG] pgmap v1465: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:56 smithi193 bash[28559]: cluster 2023-11-17T07:34:55.490866+0000 mgr.smithi097.swvonl (mgr.14180) 2084 : cluster [DBG] pgmap v1466: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:34:59 smithi193 bash[28559]: cluster 2023-11-17T07:34:57.491835+0000 mgr.smithi097.swvonl (mgr.14180) 2085 : cluster [DBG] pgmap v1467: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:01 smithi193 bash[28559]: cluster 2023-11-17T07:34:59.492977+0000 mgr.smithi097.swvonl (mgr.14180) 2086 : cluster [DBG] pgmap v1468: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:03 smithi193 bash[28559]: cluster 2023-11-17T07:35:01.494318+0000 mgr.smithi097.swvonl (mgr.14180) 2087 : cluster [DBG] pgmap v1469: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:05 smithi193 bash[28559]: cluster 2023-11-17T07:35:03.495310+0000 mgr.smithi097.swvonl (mgr.14180) 2088 : cluster [DBG] pgmap v1470: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:07 smithi193 bash[28559]: cluster 2023-11-17T07:35:05.496646+0000 mgr.smithi097.swvonl (mgr.14180) 2089 : cluster [DBG] pgmap v1471: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:09 smithi193 bash[28559]: cluster 2023-11-17T07:35:07.497620+0000 mgr.smithi097.swvonl (mgr.14180) 2090 : cluster [DBG] pgmap v1472: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.286 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:11 smithi193 bash[28559]: cluster 2023-11-17T07:35:09.498686+0000 mgr.smithi097.swvonl (mgr.14180) 2091 : cluster [DBG] pgmap v1473: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:13 smithi193 bash[28559]: cluster 2023-11-17T07:35:11.500134+0000 mgr.smithi097.swvonl (mgr.14180) 2092 : cluster [DBG] pgmap v1474: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:15 smithi193 bash[28559]: cluster 2023-11-17T07:35:13.501153+0000 mgr.smithi097.swvonl (mgr.14180) 2093 : cluster [DBG] pgmap v1475: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:17 smithi193 bash[28559]: cluster 2023-11-17T07:35:15.502575+0000 mgr.smithi097.swvonl (mgr.14180) 2094 : cluster [DBG] pgmap v1476: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:19 smithi193 bash[28559]: cluster 2023-11-17T07:35:17.503539+0000 mgr.smithi097.swvonl (mgr.14180) 2095 : cluster [DBG] pgmap v1477: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:21 smithi193 bash[28559]: cluster 2023-11-17T07:35:19.504620+0000 mgr.smithi097.swvonl (mgr.14180) 2096 : cluster [DBG] pgmap v1478: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:23 smithi193 bash[28559]: cluster 2023-11-17T07:35:21.506010+0000 mgr.smithi097.swvonl (mgr.14180) 2097 : cluster [DBG] pgmap v1479: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:25 smithi193 bash[28559]: cluster 2023-11-17T07:35:23.507035+0000 mgr.smithi097.swvonl (mgr.14180) 2098 : cluster [DBG] pgmap v1480: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:27 smithi193 bash[28559]: cluster 2023-11-17T07:35:25.508419+0000 mgr.smithi097.swvonl (mgr.14180) 2099 : cluster [DBG] pgmap v1481: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:29 smithi193 bash[28559]: cluster 2023-11-17T07:35:27.509331+0000 mgr.smithi097.swvonl (mgr.14180) 2100 : cluster [DBG] pgmap v1482: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.287 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:31 smithi193 bash[28559]: cluster 2023-11-17T07:35:29.510441+0000 mgr.smithi097.swvonl (mgr.14180) 2101 : cluster [DBG] pgmap v1483: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.480094+0000 mon.smithi097 (mon.0) 1200 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.779195+0000 mon.smithi097 (mon.0) 1201 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.780847+0000 mon.smithi097 (mon.0) 1202 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.788333+0000 mon.smithi097 (mon.0) 1203 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.794961+0000 mon.smithi097 (mon.0) 1204 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:32 smithi193 bash[28559]: audit 2023-11-17T07:35:31.808759+0000 mon.smithi097 (mon.0) 1205 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:33 smithi193 bash[28559]: cluster 2023-11-17T07:35:31.511702+0000 mgr.smithi097.swvonl (mgr.14180) 2102 : cluster [DBG] pgmap v1484: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:33 smithi193 bash[28559]: cluster 2023-11-17T07:35:31.789195+0000 mgr.smithi097.swvonl (mgr.14180) 2103 : cluster [DBG] pgmap v1485: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:33 smithi193 bash[28559]: cephadm 2023-11-17T07:35:31.814238+0000 mgr.smithi097.swvonl (mgr.14180) 2104 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.fwbuio on smithi193 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:33 smithi193 bash[28559]: cluster 2023-11-17T07:35:32.089586+0000 mon.smithi097 (mon.0) 1206 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:33 smithi193 bash[28559]: cluster 2023-11-17T07:35:32.089640+0000 mon.smithi097 (mon.0) 1207 : cluster [INF] Cluster is now healthy 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:35 smithi193 bash[28559]: cluster 2023-11-17T07:35:33.789979+0000 mgr.smithi097.swvonl (mgr.14180) 2105 : cluster [DBG] pgmap v1486: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:36 smithi193 bash[28559]: audit 2023-11-17T07:35:35.996275+0000 mon.smithi097 (mon.0) 1208 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.fwbuio"}]: dispatch 2023-11-17T07:35:44.288 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: cluster 2023-11-17T07:35:35.791126+0000 mgr.smithi097.swvonl (mgr.14180) 2106 : cluster [DBG] pgmap v1487: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: cephadm 2023-11-17T07:35:35.994950+0000 mgr.smithi097.swvonl (mgr.14180) 2107 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.289 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.290 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.291 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: cephadm 2023-11-17T07:35:35.995512+0000 mgr.smithi097.swvonl (mgr.14180) 2108 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: cephadm 2023-11-17T07:35:35.998329+0000 mgr.smithi097.swvonl (mgr.14180) 2109 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.292 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:37 smithi193 bash[28559]: cephadm 2023-11-17T07:35:36.003109+0000 mgr.smithi097.swvonl (mgr.14180) 2110 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aoqszn on smithi097 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:39 smithi193 bash[28559]: cluster 2023-11-17T07:35:37.791874+0000 mgr.smithi097.swvonl (mgr.14180) 2111 : cluster [DBG] pgmap v1488: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:40 smithi193 bash[28559]: audit 2023-11-17T07:35:39.751676+0000 mon.smithi097 (mon.0) 1209 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aoqszn"}]: dispatch 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:40 smithi193 bash[28559]: audit 2023-11-17T07:35:40.060053+0000 mon.smithi097 (mon.0) 1210 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.330 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cephadm 2023-11-17T07:35:39.750769+0000 mgr.smithi097.swvonl (mgr.14180) 2112 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:35:44.331 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.332 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cephadm 2023-11-17T07:35:39.750986+0000 mgr.smithi097.swvonl (mgr.14180) 2113 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cephadm 2023-11-17T07:35:39.753159+0000 mgr.smithi097.swvonl (mgr.14180) 2114 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.333 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cephadm 2023-11-17T07:35:39.755119+0000 mgr.smithi097.swvonl (mgr.14180) 2115 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cephadm 2023-11-17T07:35:39.756615+0000 mgr.smithi097.swvonl (mgr.14180) 2116 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cluster 2023-11-17T07:35:39.758597+0000 mgr.smithi097.swvonl (mgr.14180) 2117 : cluster [DBG] pgmap v1489: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.334 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:41 smithi193 bash[28559]: cluster 2023-11-17T07:35:40.125525+0000 mon.smithi097 (mon.0) 1211 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:37 smithi097 bash[24958]: audit 2023-11-17T07:33:36.697621+0000 mgr.smithi097.swvonl (mgr.14180) 2032 : audit [DBG] from='client.15520 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:37 smithi097 bash[24958]: cluster 2023-11-17T07:33:36.855690+0000 mgr.smithi097.swvonl (mgr.14180) 2033 : cluster [DBG] pgmap v1425: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:39 smithi097 bash[24958]: cluster 2023-11-17T07:33:38.856932+0000 mgr.smithi097.swvonl (mgr.14180) 2034 : cluster [DBG] pgmap v1426: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:42 smithi097 bash[24958]: cluster 2023-11-17T07:33:40.858382+0000 mgr.smithi097.swvonl (mgr.14180) 2035 : cluster [DBG] pgmap v1427: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:44 smithi097 bash[24958]: cluster 2023-11-17T07:33:42.859466+0000 mgr.smithi097.swvonl (mgr.14180) 2036 : cluster [DBG] pgmap v1428: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:46 smithi097 bash[24958]: cluster 2023-11-17T07:33:44.860966+0000 mgr.smithi097.swvonl (mgr.14180) 2037 : cluster [DBG] pgmap v1429: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:48 smithi097 bash[24958]: cluster 2023-11-17T07:33:46.862233+0000 mgr.smithi097.swvonl (mgr.14180) 2038 : cluster [DBG] pgmap v1430: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:50 smithi097 bash[24958]: cluster 2023-11-17T07:33:48.863434+0000 mgr.smithi097.swvonl (mgr.14180) 2039 : cluster [DBG] pgmap v1431: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.335 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:52 smithi097 bash[24958]: cluster 2023-11-17T07:33:50.864773+0000 mgr.smithi097.swvonl (mgr.14180) 2040 : cluster [DBG] pgmap v1432: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:54 smithi097 bash[24958]: cluster 2023-11-17T07:33:52.865910+0000 mgr.smithi097.swvonl (mgr.14180) 2041 : cluster [DBG] pgmap v1433: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:56 smithi097 bash[24958]: cluster 2023-11-17T07:33:54.867290+0000 mgr.smithi097.swvonl (mgr.14180) 2042 : cluster [DBG] pgmap v1434: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:33:58 smithi097 bash[24958]: cluster 2023-11-17T07:33:56.868368+0000 mgr.smithi097.swvonl (mgr.14180) 2043 : cluster [DBG] pgmap v1435: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.336 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:00 smithi097 bash[24958]: cluster 2023-11-17T07:33:58.869534+0000 mgr.smithi097.swvonl (mgr.14180) 2044 : cluster [DBG] pgmap v1436: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:02 smithi097 bash[24958]: cluster 2023-11-17T07:34:00.870830+0000 mgr.smithi097.swvonl (mgr.14180) 2045 : cluster [DBG] pgmap v1437: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:04 smithi097 bash[24958]: cluster 2023-11-17T07:34:02.871777+0000 mgr.smithi097.swvonl (mgr.14180) 2046 : cluster [DBG] pgmap v1438: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:06 smithi097 bash[24958]: cluster 2023-11-17T07:34:04.873176+0000 mgr.smithi097.swvonl (mgr.14180) 2047 : cluster [DBG] pgmap v1439: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:08 smithi097 bash[24958]: cluster 2023-11-17T07:34:06.874283+0000 mgr.smithi097.swvonl (mgr.14180) 2048 : cluster [DBG] pgmap v1440: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:10 smithi097 bash[24958]: cluster 2023-11-17T07:34:08.875292+0000 mgr.smithi097.swvonl (mgr.14180) 2049 : cluster [DBG] pgmap v1441: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:12 smithi097 bash[24958]: cluster 2023-11-17T07:34:10.876638+0000 mgr.smithi097.swvonl (mgr.14180) 2050 : cluster [DBG] pgmap v1442: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:14 smithi097 bash[24958]: cluster 2023-11-17T07:34:12.877747+0000 mgr.smithi097.swvonl (mgr.14180) 2051 : cluster [DBG] pgmap v1443: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:16 smithi097 bash[24958]: cluster 2023-11-17T07:34:14.879176+0000 mgr.smithi097.swvonl (mgr.14180) 2052 : cluster [DBG] pgmap v1444: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:18 smithi097 bash[24958]: cluster 2023-11-17T07:34:16.880189+0000 mgr.smithi097.swvonl (mgr.14180) 2053 : cluster [DBG] pgmap v1445: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:20 smithi097 bash[24958]: cluster 2023-11-17T07:34:18.881402+0000 mgr.smithi097.swvonl (mgr.14180) 2054 : cluster [DBG] pgmap v1446: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:22 smithi097 bash[24958]: cluster 2023-11-17T07:34:20.882919+0000 mgr.smithi097.swvonl (mgr.14180) 2055 : cluster [DBG] pgmap v1447: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:23 smithi097 bash[24958]: audit 2023-11-17T07:34:22.852234+0000 mon.smithi097 (mon.0) 1186 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: cluster 2023-11-17T07:34:22.883696+0000 mgr.smithi097.swvonl (mgr.14180) 2056 : cluster [DBG] pgmap v1448: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: audit 2023-11-17T07:34:23.160843+0000 mon.smithi097 (mon.0) 1187 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: audit 2023-11-17T07:34:23.162082+0000 mon.smithi097 (mon.0) 1188 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: audit 2023-11-17T07:34:23.169488+0000 mon.smithi097 (mon.0) 1189 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: audit 2023-11-17T07:34:23.178043+0000 mon.smithi097 (mon.0) 1190 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:24 smithi097 bash[24958]: audit 2023-11-17T07:34:23.191499+0000 mon.smithi097 (mon.0) 1191 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.374 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:25 smithi097 bash[24958]: cluster 2023-11-17T07:34:23.170322+0000 mgr.smithi097.swvonl (mgr.14180) 2057 : cluster [DBG] pgmap v1449: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:25 smithi097 bash[24958]: cephadm 2023-11-17T07:34:23.194756+0000 mgr.smithi097.swvonl (mgr.14180) 2058 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.tskdgs on smithi193 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:25 smithi097 bash[24958]: cluster 2023-11-17T07:34:24.166305+0000 mon.smithi097 (mon.0) 1192 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:25 smithi097 bash[24958]: cluster 2023-11-17T07:34:24.166356+0000 mon.smithi097 (mon.0) 1193 : cluster [INF] Cluster is now healthy 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:27 smithi097 bash[24958]: cluster 2023-11-17T07:34:25.171336+0000 mgr.smithi097.swvonl (mgr.14180) 2059 : cluster [DBG] pgmap v1450: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:27 smithi097 bash[24958]: audit 2023-11-17T07:34:26.571662+0000 mon.smithi097 (mon.0) 1194 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:28 smithi097 bash[24958]: audit 2023-11-17T07:34:27.283400+0000 mon.smithi097 (mon.0) 1195 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.tskdgs"}]: dispatch 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: cluster 2023-11-17T07:34:27.172258+0000 mgr.smithi097.swvonl (mgr.14180) 2060 : cluster [DBG] pgmap v1451: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: cephadm 2023-11-17T07:34:27.282287+0000 mgr.smithi097.swvonl (mgr.14180) 2061 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.375 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:35:44.376 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: cephadm 2023-11-17T07:34:27.282792+0000 mgr.smithi097.swvonl (mgr.14180) 2062 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.377 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: cephadm 2023-11-17T07:34:27.284995+0000 mgr.smithi097.swvonl (mgr.14180) 2063 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.tskdgs on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-tskdgs 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.tskdgs 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.tskdgs ... 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:29 smithi097 bash[24958]: cephadm 2023-11-17T07:34:27.288866+0000 mgr.smithi097.swvonl (mgr.14180) 2064 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lzgrra on smithi097 2023-11-17T07:35:44.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:31 smithi097 bash[24958]: cluster 2023-11-17T07:34:29.173008+0000 mgr.smithi097.swvonl (mgr.14180) 2065 : cluster [DBG] pgmap v1452: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:32 smithi097 bash[24958]: audit 2023-11-17T07:34:31.468628+0000 mon.smithi097 (mon.0) 1196 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lzgrra"}]: dispatch 2023-11-17T07:35:44.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cluster 2023-11-17T07:34:31.173844+0000 mgr.smithi097.swvonl (mgr.14180) 2066 : cluster [DBG] pgmap v1453: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.379 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cephadm 2023-11-17T07:34:31.467337+0000 mgr.smithi097.swvonl (mgr.14180) 2067 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.397 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.398 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cephadm 2023-11-17T07:34:31.467848+0000 mgr.smithi097.swvonl (mgr.14180) 2068 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cephadm 2023-11-17T07:34:31.470202+0000 mgr.smithi097.swvonl (mgr.14180) 2069 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lzgrra on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lzgrra 2023-11-17T07:35:44.399 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lzgrra 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lzgrra ... 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cephadm 2023-11-17T07:34:31.473033+0000 mgr.smithi097.swvonl (mgr.14180) 2070 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.oulsqb on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cephadm 2023-11-17T07:34:31.475312+0000 mgr.smithi097.swvonl (mgr.14180) 2071 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cluster 2023-11-17T07:34:31.476465+0000 mgr.smithi097.swvonl (mgr.14180) 2072 : cluster [DBG] pgmap v1454: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 238 B/s rd, 0 op/s 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:33 smithi097 bash[24958]: cluster 2023-11-17T07:34:32.154298+0000 mon.smithi097 (mon.0) 1197 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:35:44.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:34 smithi097 bash[24958]: cluster 2023-11-17T07:34:33.477508+0000 mgr.smithi097.swvonl (mgr.14180) 2073 : cluster [DBG] pgmap v1455: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:36 smithi097 bash[24958]: audit 2023-11-17T07:34:35.009348+0000 mon.smithi097 (mon.0) 1198 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:37 smithi097 bash[24958]: cluster 2023-11-17T07:34:35.478799+0000 mgr.smithi097.swvonl (mgr.14180) 2074 : cluster [DBG] pgmap v1456: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:39 smithi097 bash[24958]: cluster 2023-11-17T07:34:37.479782+0000 mgr.smithi097.swvonl (mgr.14180) 2075 : cluster [DBG] pgmap v1457: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:41 smithi097 bash[24958]: cluster 2023-11-17T07:34:39.480888+0000 mgr.smithi097.swvonl (mgr.14180) 2076 : cluster [DBG] pgmap v1458: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:42 smithi097 bash[24958]: cluster 2023-11-17T07:34:41.482236+0000 mgr.smithi097.swvonl (mgr.14180) 2077 : cluster [DBG] pgmap v1459: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:42 smithi097 bash[24958]: audit 2023-11-17T07:34:41.571664+0000 mon.smithi097 (mon.0) 1199 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:44 smithi097 bash[24958]: cluster 2023-11-17T07:34:43.483292+0000 mgr.smithi097.swvonl (mgr.14180) 2078 : cluster [DBG] pgmap v1460: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:46 smithi097 bash[24958]: cluster 2023-11-17T07:34:45.484759+0000 mgr.smithi097.swvonl (mgr.14180) 2079 : cluster [DBG] pgmap v1461: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:48 smithi097 bash[24958]: cluster 2023-11-17T07:34:47.485702+0000 mgr.smithi097.swvonl (mgr.14180) 2080 : cluster [DBG] pgmap v1462: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:50 smithi097 bash[24958]: cluster 2023-11-17T07:34:49.486804+0000 mgr.smithi097.swvonl (mgr.14180) 2081 : cluster [DBG] pgmap v1463: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:52 smithi097 bash[24958]: cluster 2023-11-17T07:34:51.488261+0000 mgr.smithi097.swvonl (mgr.14180) 2082 : cluster [DBG] pgmap v1464: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:54 smithi097 bash[24958]: cluster 2023-11-17T07:34:53.489473+0000 mgr.smithi097.swvonl (mgr.14180) 2083 : cluster [DBG] pgmap v1465: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.401 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:56 smithi097 bash[24958]: cluster 2023-11-17T07:34:55.490866+0000 mgr.smithi097.swvonl (mgr.14180) 2084 : cluster [DBG] pgmap v1466: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:34:58 smithi097 bash[24958]: cluster 2023-11-17T07:34:57.491835+0000 mgr.smithi097.swvonl (mgr.14180) 2085 : cluster [DBG] pgmap v1467: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:01 smithi097 bash[24958]: cluster 2023-11-17T07:34:59.492977+0000 mgr.smithi097.swvonl (mgr.14180) 2086 : cluster [DBG] pgmap v1468: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:03 smithi097 bash[24958]: cluster 2023-11-17T07:35:01.494318+0000 mgr.smithi097.swvonl (mgr.14180) 2087 : cluster [DBG] pgmap v1469: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:05 smithi097 bash[24958]: cluster 2023-11-17T07:35:03.495310+0000 mgr.smithi097.swvonl (mgr.14180) 2088 : cluster [DBG] pgmap v1470: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:07 smithi097 bash[24958]: cluster 2023-11-17T07:35:05.496646+0000 mgr.smithi097.swvonl (mgr.14180) 2089 : cluster [DBG] pgmap v1471: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:09 smithi097 bash[24958]: cluster 2023-11-17T07:35:07.497620+0000 mgr.smithi097.swvonl (mgr.14180) 2090 : cluster [DBG] pgmap v1472: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:11 smithi097 bash[24958]: cluster 2023-11-17T07:35:09.498686+0000 mgr.smithi097.swvonl (mgr.14180) 2091 : cluster [DBG] pgmap v1473: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:13 smithi097 bash[24958]: cluster 2023-11-17T07:35:11.500134+0000 mgr.smithi097.swvonl (mgr.14180) 2092 : cluster [DBG] pgmap v1474: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:15 smithi097 bash[24958]: cluster 2023-11-17T07:35:13.501153+0000 mgr.smithi097.swvonl (mgr.14180) 2093 : cluster [DBG] pgmap v1475: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:17 smithi097 bash[24958]: cluster 2023-11-17T07:35:15.502575+0000 mgr.smithi097.swvonl (mgr.14180) 2094 : cluster [DBG] pgmap v1476: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:19 smithi097 bash[24958]: cluster 2023-11-17T07:35:17.503539+0000 mgr.smithi097.swvonl (mgr.14180) 2095 : cluster [DBG] pgmap v1477: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.402 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:21 smithi097 bash[24958]: cluster 2023-11-17T07:35:19.504620+0000 mgr.smithi097.swvonl (mgr.14180) 2096 : cluster [DBG] pgmap v1478: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:23 smithi097 bash[24958]: cluster 2023-11-17T07:35:21.506010+0000 mgr.smithi097.swvonl (mgr.14180) 2097 : cluster [DBG] pgmap v1479: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:25 smithi097 bash[24958]: cluster 2023-11-17T07:35:23.507035+0000 mgr.smithi097.swvonl (mgr.14180) 2098 : cluster [DBG] pgmap v1480: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:27 smithi097 bash[24958]: cluster 2023-11-17T07:35:25.508419+0000 mgr.smithi097.swvonl (mgr.14180) 2099 : cluster [DBG] pgmap v1481: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:29 smithi097 bash[24958]: cluster 2023-11-17T07:35:27.509331+0000 mgr.smithi097.swvonl (mgr.14180) 2100 : cluster [DBG] pgmap v1482: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:31 smithi097 bash[24958]: cluster 2023-11-17T07:35:29.510441+0000 mgr.smithi097.swvonl (mgr.14180) 2101 : cluster [DBG] pgmap v1483: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.480094+0000 mon.smithi097 (mon.0) 1200 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.779195+0000 mon.smithi097 (mon.0) 1201 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.780847+0000 mon.smithi097 (mon.0) 1202 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.788333+0000 mon.smithi097 (mon.0) 1203 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.794961+0000 mon.smithi097 (mon.0) 1204 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:35:44.403 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:32 smithi097 bash[24958]: audit 2023-11-17T07:35:31.808759+0000 mon.smithi097 (mon.0) 1205 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:33 smithi097 bash[24958]: cluster 2023-11-17T07:35:31.511702+0000 mgr.smithi097.swvonl (mgr.14180) 2102 : cluster [DBG] pgmap v1484: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:33 smithi097 bash[24958]: cluster 2023-11-17T07:35:31.789195+0000 mgr.smithi097.swvonl (mgr.14180) 2103 : cluster [DBG] pgmap v1485: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:33 smithi097 bash[24958]: cephadm 2023-11-17T07:35:31.814238+0000 mgr.smithi097.swvonl (mgr.14180) 2104 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.fwbuio on smithi193 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:33 smithi097 bash[24958]: cluster 2023-11-17T07:35:32.089586+0000 mon.smithi097 (mon.0) 1206 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:33 smithi097 bash[24958]: cluster 2023-11-17T07:35:32.089640+0000 mon.smithi097 (mon.0) 1207 : cluster [INF] Cluster is now healthy 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:35 smithi097 bash[24958]: cluster 2023-11-17T07:35:33.789979+0000 mgr.smithi097.swvonl (mgr.14180) 2105 : cluster [DBG] pgmap v1486: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:36 smithi097 bash[24958]: audit 2023-11-17T07:35:35.996275+0000 mon.smithi097 (mon.0) 1208 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.fwbuio"}]: dispatch 2023-11-17T07:35:44.438 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: cluster 2023-11-17T07:35:35.791126+0000 mgr.smithi097.swvonl (mgr.14180) 2106 : cluster [DBG] pgmap v1487: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: cephadm 2023-11-17T07:35:35.994950+0000 mgr.smithi097.swvonl (mgr.14180) 2107 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.439 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.440 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: cephadm 2023-11-17T07:35:35.995512+0000 mgr.smithi097.swvonl (mgr.14180) 2108 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: cephadm 2023-11-17T07:35:35.998329+0000 mgr.smithi097.swvonl (mgr.14180) 2109 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio 2023-11-17T07:35:44.441 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.fwbuio ... 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:37 smithi097 bash[24958]: cephadm 2023-11-17T07:35:36.003109+0000 mgr.smithi097.swvonl (mgr.14180) 2110 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.aoqszn on smithi097 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:39 smithi097 bash[24958]: cluster 2023-11-17T07:35:37.791874+0000 mgr.smithi097.swvonl (mgr.14180) 2111 : cluster [DBG] pgmap v1488: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:40 smithi097 bash[24958]: audit 2023-11-17T07:35:39.751676+0000 mon.smithi097 (mon.0) 1209 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.aoqszn"}]: dispatch 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:40 smithi097 bash[24958]: audit 2023-11-17T07:35:40.060053+0000 mon.smithi097 (mon.0) 1210 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cephadm 2023-11-17T07:35:39.750769+0000 mgr.smithi097.swvonl (mgr.14180) 2112 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.442 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:35:44.443 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.444 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cephadm 2023-11-17T07:35:39.750986+0000 mgr.smithi097.swvonl (mgr.14180) 2113 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cephadm 2023-11-17T07:35:39.753159+0000 mgr.smithi097.swvonl (mgr.14180) 2114 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.aoqszn ... 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:35:44.445 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cephadm 2023-11-17T07:35:39.755119+0000 mgr.smithi097.swvonl (mgr.14180) 2115 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cephadm 2023-11-17T07:35:39.756615+0000 mgr.smithi097.swvonl (mgr.14180) 2116 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:35:44.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cluster 2023-11-17T07:35:39.758597+0000 mgr.smithi097.swvonl (mgr.14180) 2117 : cluster [DBG] pgmap v1489: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:41 smithi097 bash[24958]: cluster 2023-11-17T07:35:40.125525+0000 mon.smithi097 (mon.0) 1211 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:35:44.446 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:43 smithi097 bash[24958]: cluster 2023-11-17T07:35:41.759753+0000 mgr.smithi097.swvonl (mgr.14180) 2118 : cluster [DBG] pgmap v1490: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:44.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:35:44.447 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:43 smithi193 bash[28559]: cluster 2023-11-17T07:35:41.759753+0000 mgr.smithi097.swvonl (mgr.14180) 2118 : cluster [DBG] pgmap v1490: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:35:45.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:45 smithi097 bash[24958]: cluster 2023-11-17T07:35:43.760856+0000 mgr.smithi097.swvonl (mgr.14180) 2119 : cluster [DBG] pgmap v1491: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:45.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:45 smithi193 bash[28559]: cluster 2023-11-17T07:35:43.760856+0000 mgr.smithi097.swvonl (mgr.14180) 2119 : cluster [DBG] pgmap v1491: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:47.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:47 smithi097 bash[24958]: cluster 2023-11-17T07:35:45.762202+0000 mgr.smithi097.swvonl (mgr.14180) 2120 : cluster [DBG] pgmap v1492: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:47.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:47 smithi193 bash[28559]: cluster 2023-11-17T07:35:45.762202+0000 mgr.smithi097.swvonl (mgr.14180) 2120 : cluster [DBG] pgmap v1492: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:48.774 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:35:48.774 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:35:49.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:49 smithi097 bash[24958]: cluster 2023-11-17T07:35:47.763153+0000 mgr.smithi097.swvonl (mgr.14180) 2121 : cluster [DBG] pgmap v1493: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:49.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:49 smithi193 bash[28559]: cluster 2023-11-17T07:35:47.763153+0000 mgr.smithi097.swvonl (mgr.14180) 2121 : cluster [DBG] pgmap v1493: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:49.703 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:35:50.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:50 smithi097 bash[24958]: audit 2023-11-17T07:35:48.763201+0000 mgr.smithi097.swvonl (mgr.14180) 2122 : audit [DBG] from='client.15524 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:50.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:50 smithi193 bash[28559]: audit 2023-11-17T07:35:48.763201+0000 mgr.smithi097.swvonl (mgr.14180) 2122 : audit [DBG] from='client.15524 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:50.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:35:51.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:51 smithi097 bash[24958]: cluster 2023-11-17T07:35:49.764291+0000 mgr.smithi097.swvonl (mgr.14180) 2123 : cluster [DBG] pgmap v1494: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:51.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:51 smithi193 bash[28559]: cluster 2023-11-17T07:35:49.764291+0000 mgr.smithi097.swvonl (mgr.14180) 2123 : cluster [DBG] pgmap v1494: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:35:53.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:53 smithi097 bash[24958]: cluster 2023-11-17T07:35:51.765540+0000 mgr.smithi097.swvonl (mgr.14180) 2124 : cluster [DBG] pgmap v1495: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:53.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:53 smithi193 bash[28559]: cluster 2023-11-17T07:35:51.765540+0000 mgr.smithi097.swvonl (mgr.14180) 2124 : cluster [DBG] pgmap v1495: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:35:54.880 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:35:54.881 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:35:55.236 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:55 smithi097 bash[24958]: cluster 2023-11-17T07:35:53.766679+0000 mgr.smithi097.swvonl (mgr.14180) 2125 : cluster [DBG] pgmap v1496: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:55.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:55 smithi193 bash[28559]: cluster 2023-11-17T07:35:53.766679+0000 mgr.smithi097.swvonl (mgr.14180) 2125 : cluster [DBG] pgmap v1496: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:55.901 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:35:56.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:56 smithi097 bash[24958]: audit 2023-11-17T07:35:54.866940+0000 mgr.smithi097.swvonl (mgr.14180) 2126 : audit [DBG] from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:56.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:56 smithi193 bash[28559]: audit 2023-11-17T07:35:54.866940+0000 mgr.smithi097.swvonl (mgr.14180) 2126 : audit [DBG] from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:35:56.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:35:57.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:57 smithi097 bash[24958]: cluster 2023-11-17T07:35:55.768028+0000 mgr.smithi097.swvonl (mgr.14180) 2127 : cluster [DBG] pgmap v1497: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:57.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:57 smithi193 bash[28559]: cluster 2023-11-17T07:35:55.768028+0000 mgr.smithi097.swvonl (mgr.14180) 2127 : cluster [DBG] pgmap v1497: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:58.465 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:35:58 smithi097 bash[24958]: cluster 2023-11-17T07:35:57.769018+0000 mgr.smithi097.swvonl (mgr.14180) 2128 : cluster [DBG] pgmap v1498: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:35:58.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:35:58 smithi193 bash[28559]: cluster 2023-11-17T07:35:57.769018+0000 mgr.smithi097.swvonl (mgr.14180) 2128 : cluster [DBG] pgmap v1498: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:00.937 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:00.938 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:01.244 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:00 smithi097 bash[24958]: cluster 2023-11-17T07:35:59.770297+0000 mgr.smithi097.swvonl (mgr.14180) 2129 : cluster [DBG] pgmap v1499: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:01.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:00 smithi193 bash[28559]: cluster 2023-11-17T07:35:59.770297+0000 mgr.smithi097.swvonl (mgr.14180) 2129 : cluster [DBG] pgmap v1499: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:01.890 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:02.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:01 smithi097 bash[24958]: audit 2023-11-17T07:36:00.927038+0000 mgr.smithi097.swvonl (mgr.14180) 2130 : audit [DBG] from='client.15532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:02.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:01 smithi193 bash[28559]: audit 2023-11-17T07:36:00.927038+0000 mgr.smithi097.swvonl (mgr.14180) 2130 : audit [DBG] from='client.15532 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:02.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:03.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:03 smithi097 bash[24958]: cluster 2023-11-17T07:36:01.771455+0000 mgr.smithi097.swvonl (mgr.14180) 2131 : cluster [DBG] pgmap v1500: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:03.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:03 smithi193 bash[28559]: cluster 2023-11-17T07:36:01.771455+0000 mgr.smithi097.swvonl (mgr.14180) 2131 : cluster [DBG] pgmap v1500: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:05.284 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:05 smithi193 bash[28559]: cluster 2023-11-17T07:36:03.772566+0000 mgr.smithi097.swvonl (mgr.14180) 2132 : cluster [DBG] pgmap v1501: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:05.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:05 smithi097 bash[24958]: cluster 2023-11-17T07:36:03.772566+0000 mgr.smithi097.swvonl (mgr.14180) 2132 : cluster [DBG] pgmap v1501: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:06.817 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:06.818 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:07.142 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:07 smithi097 bash[24958]: cluster 2023-11-17T07:36:05.773491+0000 mgr.smithi097.swvonl (mgr.14180) 2133 : cluster [DBG] pgmap v1502: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:07.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:07 smithi193 bash[28559]: cluster 2023-11-17T07:36:05.773491+0000 mgr.smithi097.swvonl (mgr.14180) 2133 : cluster [DBG] pgmap v1502: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:07.760 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:08.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:08 smithi097 bash[24958]: audit 2023-11-17T07:36:06.803569+0000 mgr.smithi097.swvonl (mgr.14180) 2134 : audit [DBG] from='client.15536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:08.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:08 smithi193 bash[28559]: audit 2023-11-17T07:36:06.803569+0000 mgr.smithi097.swvonl (mgr.14180) 2134 : audit [DBG] from='client.15536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:08.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:09.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:09 smithi097 bash[24958]: cluster 2023-11-17T07:36:07.774504+0000 mgr.smithi097.swvonl (mgr.14180) 2135 : cluster [DBG] pgmap v1503: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:09.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:09 smithi193 bash[28559]: cluster 2023-11-17T07:36:07.774504+0000 mgr.smithi097.swvonl (mgr.14180) 2135 : cluster [DBG] pgmap v1503: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:11 smithi097 bash[24958]: cluster 2023-11-17T07:36:09.775629+0000 mgr.smithi097.swvonl (mgr.14180) 2136 : cluster [DBG] pgmap v1504: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:11.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:11 smithi193 bash[28559]: cluster 2023-11-17T07:36:09.775629+0000 mgr.smithi097.swvonl (mgr.14180) 2136 : cluster [DBG] pgmap v1504: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:12.913 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:12.913 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:13.199 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:13 smithi097 bash[24958]: cluster 2023-11-17T07:36:11.776737+0000 mgr.smithi097.swvonl (mgr.14180) 2137 : cluster [DBG] pgmap v1505: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:13.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:13 smithi193 bash[28559]: cluster 2023-11-17T07:36:11.776737+0000 mgr.smithi097.swvonl (mgr.14180) 2137 : cluster [DBG] pgmap v1505: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:13.852 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:14.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:14 smithi097 bash[24958]: audit 2023-11-17T07:36:12.898994+0000 mgr.smithi097.swvonl (mgr.14180) 2138 : audit [DBG] from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:14.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:14 smithi193 bash[28559]: audit 2023-11-17T07:36:12.898994+0000 mgr.smithi097.swvonl (mgr.14180) 2138 : audit [DBG] from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:14.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:15.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:15 smithi097 bash[24958]: cluster 2023-11-17T07:36:13.777688+0000 mgr.smithi097.swvonl (mgr.14180) 2139 : cluster [DBG] pgmap v1506: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:15.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:15 smithi193 bash[28559]: cluster 2023-11-17T07:36:13.777688+0000 mgr.smithi097.swvonl (mgr.14180) 2139 : cluster [DBG] pgmap v1506: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:17.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:17 smithi097 bash[24958]: cluster 2023-11-17T07:36:15.778874+0000 mgr.smithi097.swvonl (mgr.14180) 2140 : cluster [DBG] pgmap v1507: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:17.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:17 smithi193 bash[28559]: cluster 2023-11-17T07:36:15.778874+0000 mgr.smithi097.swvonl (mgr.14180) 2140 : cluster [DBG] pgmap v1507: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:18.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:18.869 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:19.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:19 smithi097 bash[24958]: cluster 2023-11-17T07:36:17.779825+0000 mgr.smithi097.swvonl (mgr.14180) 2141 : cluster [DBG] pgmap v1508: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:19 smithi193 bash[28559]: cluster 2023-11-17T07:36:17.779825+0000 mgr.smithi097.swvonl (mgr.14180) 2141 : cluster [DBG] pgmap v1508: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:19.757 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:20.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:20 smithi097 bash[24958]: audit 2023-11-17T07:36:18.854240+0000 mgr.smithi097.swvonl (mgr.14180) 2142 : audit [DBG] from='client.15544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:20.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:20 smithi193 bash[28559]: audit 2023-11-17T07:36:18.854240+0000 mgr.smithi097.swvonl (mgr.14180) 2142 : audit [DBG] from='client.15544 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:20.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:21.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:21 smithi097 bash[24958]: cluster 2023-11-17T07:36:19.781063+0000 mgr.smithi097.swvonl (mgr.14180) 2143 : cluster [DBG] pgmap v1509: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:21 smithi193 bash[28559]: cluster 2023-11-17T07:36:19.781063+0000 mgr.smithi097.swvonl (mgr.14180) 2143 : cluster [DBG] pgmap v1509: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:23.345 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:23 smithi097 bash[24958]: cluster 2023-11-17T07:36:21.781738+0000 mgr.smithi097.swvonl (mgr.14180) 2144 : cluster [DBG] pgmap v1510: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:23.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:23 smithi193 bash[28559]: cluster 2023-11-17T07:36:21.781738+0000 mgr.smithi097.swvonl (mgr.14180) 2144 : cluster [DBG] pgmap v1510: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:24.831 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:24.831 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:25.148 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:25 smithi097 bash[24958]: cluster 2023-11-17T07:36:23.782783+0000 mgr.smithi097.swvonl (mgr.14180) 2145 : cluster [DBG] pgmap v1511: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:25 smithi193 bash[28559]: cluster 2023-11-17T07:36:23.782783+0000 mgr.smithi097.swvonl (mgr.14180) 2145 : cluster [DBG] pgmap v1511: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:25.828 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:26.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:26 smithi097 bash[24958]: audit 2023-11-17T07:36:24.819387+0000 mgr.smithi097.swvonl (mgr.14180) 2146 : audit [DBG] from='client.15548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:26.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:26 smithi193 bash[28559]: audit 2023-11-17T07:36:24.819387+0000 mgr.smithi097.swvonl (mgr.14180) 2146 : audit [DBG] from='client.15548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:26.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:27.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:27 smithi097 bash[24958]: cluster 2023-11-17T07:36:25.784056+0000 mgr.smithi097.swvonl (mgr.14180) 2147 : cluster [DBG] pgmap v1512: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:27 smithi193 bash[28559]: cluster 2023-11-17T07:36:25.784056+0000 mgr.smithi097.swvonl (mgr.14180) 2147 : cluster [DBG] pgmap v1512: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:29 smithi097 bash[24958]: cluster 2023-11-17T07:36:27.784787+0000 mgr.smithi097.swvonl (mgr.14180) 2148 : cluster [DBG] pgmap v1513: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:29.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:29 smithi193 bash[28559]: cluster 2023-11-17T07:36:27.784787+0000 mgr.smithi097.swvonl (mgr.14180) 2148 : cluster [DBG] pgmap v1513: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:30.805 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:30.805 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:31.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:31 smithi097 bash[24958]: cluster 2023-11-17T07:36:29.785480+0000 mgr.smithi097.swvonl (mgr.14180) 2149 : cluster [DBG] pgmap v1514: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:31.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:31 smithi193 bash[28559]: cluster 2023-11-17T07:36:29.785480+0000 mgr.smithi097.swvonl (mgr.14180) 2149 : cluster [DBG] pgmap v1514: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:31.757 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:32.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:32 smithi097 bash[24958]: audit 2023-11-17T07:36:30.792101+0000 mgr.smithi097.swvonl (mgr.14180) 2150 : audit [DBG] from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:32.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:32 smithi193 bash[28559]: audit 2023-11-17T07:36:30.792101+0000 mgr.smithi097.swvonl (mgr.14180) 2150 : audit [DBG] from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:32.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:33.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:33 smithi097 bash[24958]: cluster 2023-11-17T07:36:31.786763+0000 mgr.smithi097.swvonl (mgr.14180) 2151 : cluster [DBG] pgmap v1515: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:33.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:33 smithi193 bash[28559]: cluster 2023-11-17T07:36:31.786763+0000 mgr.smithi097.swvonl (mgr.14180) 2151 : cluster [DBG] pgmap v1515: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:36:35.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:35 smithi097 bash[24958]: cluster 2023-11-17T07:36:33.787971+0000 mgr.smithi097.swvonl (mgr.14180) 2152 : cluster [DBG] pgmap v1516: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:35.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:35 smithi193 bash[28559]: cluster 2023-11-17T07:36:33.787971+0000 mgr.smithi097.swvonl (mgr.14180) 2152 : cluster [DBG] pgmap v1516: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:36.753 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:36.754 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:37.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:37 smithi097 bash[24958]: cluster 2023-11-17T07:36:35.789143+0000 mgr.smithi097.swvonl (mgr.14180) 2153 : cluster [DBG] pgmap v1517: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:37.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:37 smithi193 bash[28559]: cluster 2023-11-17T07:36:35.789143+0000 mgr.smithi097.swvonl (mgr.14180) 2153 : cluster [DBG] pgmap v1517: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:37.778 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:38 smithi097 bash[24958]: audit 2023-11-17T07:36:36.742227+0000 mgr.smithi097.swvonl (mgr.14180) 2154 : audit [DBG] from='client.15556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:38.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:38 smithi193 bash[28559]: audit 2023-11-17T07:36:36.742227+0000 mgr.smithi097.swvonl (mgr.14180) 2154 : audit [DBG] from='client.15556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:38.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:39.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:39 smithi097 bash[24958]: cluster 2023-11-17T07:36:37.790260+0000 mgr.smithi097.swvonl (mgr.14180) 2155 : cluster [DBG] pgmap v1518: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:39 smithi193 bash[28559]: cluster 2023-11-17T07:36:37.790260+0000 mgr.smithi097.swvonl (mgr.14180) 2155 : cluster [DBG] pgmap v1518: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:40.386 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:39.763748+0000 mon.smithi097 (mon.0) 1212 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:36:40.386 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:40.070119+0000 mon.smithi097 (mon.0) 1213 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:36:40.386 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:40.071740+0000 mon.smithi097 (mon.0) 1214 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:36:40.387 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:40.078880+0000 mon.smithi097 (mon.0) 1215 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:40.387 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:40.086689+0000 mon.smithi097 (mon.0) 1216 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:36:40.387 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:40 smithi097 bash[24958]: audit 2023-11-17T07:36:40.099116+0000 mon.smithi097 (mon.0) 1217 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:40.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:39.763748+0000 mon.smithi097 (mon.0) 1212 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:36:40.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:40.070119+0000 mon.smithi097 (mon.0) 1213 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:36:40.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:40.071740+0000 mon.smithi097 (mon.0) 1214 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:36:40.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:40.078880+0000 mon.smithi097 (mon.0) 1215 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:40.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:40.086689+0000 mon.smithi097 (mon.0) 1216 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:36:40.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:40 smithi193 bash[28559]: audit 2023-11-17T07:36:40.099116+0000 mon.smithi097 (mon.0) 1217 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:41 smithi097 bash[24958]: cluster 2023-11-17T07:36:39.791722+0000 mgr.smithi097.swvonl (mgr.14180) 2156 : cluster [DBG] pgmap v1519: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:41 smithi097 bash[24958]: cluster 2023-11-17T07:36:40.079548+0000 mgr.smithi097.swvonl (mgr.14180) 2157 : cluster [DBG] pgmap v1520: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 397 B/s rd, 0 op/s 2023-11-17T07:36:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:41 smithi097 bash[24958]: cephadm 2023-11-17T07:36:40.101861+0000 mgr.smithi097.swvonl (mgr.14180) 2158 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kzeexl on smithi193 2023-11-17T07:36:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:41 smithi097 bash[24958]: cluster 2023-11-17T07:36:40.160792+0000 mon.smithi097 (mon.0) 1218 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:36:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:41 smithi097 bash[24958]: cluster 2023-11-17T07:36:40.160851+0000 mon.smithi097 (mon.0) 1219 : cluster [INF] Cluster is now healthy 2023-11-17T07:36:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:41 smithi193 bash[28559]: cluster 2023-11-17T07:36:39.791722+0000 mgr.smithi097.swvonl (mgr.14180) 2156 : cluster [DBG] pgmap v1519: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:36:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:41 smithi193 bash[28559]: cluster 2023-11-17T07:36:40.079548+0000 mgr.smithi097.swvonl (mgr.14180) 2157 : cluster [DBG] pgmap v1520: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 397 B/s rd, 0 op/s 2023-11-17T07:36:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:41 smithi193 bash[28559]: cephadm 2023-11-17T07:36:40.101861+0000 mgr.smithi097.swvonl (mgr.14180) 2158 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.kzeexl on smithi193 2023-11-17T07:36:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:41 smithi193 bash[28559]: cluster 2023-11-17T07:36:40.160792+0000 mon.smithi097 (mon.0) 1218 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:36:41.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:41 smithi193 bash[28559]: cluster 2023-11-17T07:36:40.160851+0000 mon.smithi097 (mon.0) 1219 : cluster [INF] Cluster is now healthy 2023-11-17T07:36:42.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:42 smithi097 bash[24958]: audit 2023-11-17T07:36:41.575611+0000 mon.smithi097 (mon.0) 1220 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:42.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:42 smithi097 bash[24958]: cluster 2023-11-17T07:36:42.080311+0000 mgr.smithi097.swvonl (mgr.14180) 2159 : cluster [DBG] pgmap v1521: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:42.890 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:42.891 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:34:31.475184Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.uiyokx on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:35.998037Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.fwbuio on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-fwbuio\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.fwbuio\nDeploy daemon haproxy.nfs.foo.smithi193.fwbuio ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.752924Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.aoqszn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-aoqszn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.aoqszn\nDeploy daemon haproxy.nfs.foo.smithi097.aoqszn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:35:39.755025Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgcwra on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:43.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:42 smithi193 bash[28559]: audit 2023-11-17T07:36:41.575611+0000 mon.smithi097 (mon.0) 1220 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:43.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:42 smithi193 bash[28559]: cluster 2023-11-17T07:36:42.080311+0000 mgr.smithi097.swvonl (mgr.14180) 2159 : cluster [DBG] pgmap v1521: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:43.856 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:44.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:44 smithi097 bash[24958]: audit 2023-11-17T07:36:42.879727+0000 mgr.smithi097.swvonl (mgr.14180) 2160 : audit [DBG] from='client.15560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:44.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:44 smithi193 bash[28559]: audit 2023-11-17T07:36:42.879727+0000 mgr.smithi097.swvonl (mgr.14180) 2160 : audit [DBG] from='client.15560 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:44.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:45.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:45 smithi097 bash[24958]: cluster 2023-11-17T07:36:44.081074+0000 mgr.smithi097.swvonl (mgr.14180) 2161 : cluster [DBG] pgmap v1522: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:45.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:45 smithi097 bash[24958]: audit 2023-11-17T07:36:44.300415+0000 mon.smithi097 (mon.0) 1221 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kzeexl"}]: dispatch 2023-11-17T07:36:45.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:45 smithi193 bash[28559]: cluster 2023-11-17T07:36:44.081074+0000 mgr.smithi097.swvonl (mgr.14180) 2161 : cluster [DBG] pgmap v1522: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:45.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:45 smithi193 bash[28559]: audit 2023-11-17T07:36:44.300415+0000 mon.smithi097 (mon.0) 1221 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.kzeexl"}]: dispatch 2023-11-17T07:36:46.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: cephadm 2023-11-17T07:36:44.299230+0000 mgr.smithi097.swvonl (mgr.14180) 2162 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:36:46.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.301 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: cephadm 2023-11-17T07:36:44.299748+0000 mgr.smithi097.swvonl (mgr.14180) 2163 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: cephadm 2023-11-17T07:36:44.301985+0000 mgr.smithi097.swvonl (mgr.14180) 2164 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.302 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.303 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:46 smithi097 bash[24958]: cephadm 2023-11-17T07:36:44.305893+0000 mgr.smithi097.swvonl (mgr.14180) 2165 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.sdmmgy on smithi097 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: cephadm 2023-11-17T07:36:44.299230+0000 mgr.smithi097.swvonl (mgr.14180) 2162 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.308 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: cephadm 2023-11-17T07:36:44.299748+0000 mgr.smithi097.swvonl (mgr.14180) 2163 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: cephadm 2023-11-17T07:36:44.301985+0000 mgr.smithi097.swvonl (mgr.14180) 2164 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.kzeexl ... 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:36:46.309 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:46.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:46.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:46.310 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:46 smithi193 bash[28559]: cephadm 2023-11-17T07:36:44.305893+0000 mgr.smithi097.swvonl (mgr.14180) 2165 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.sdmmgy on smithi097 2023-11-17T07:36:47.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:47 smithi097 bash[24958]: cluster 2023-11-17T07:36:46.082165+0000 mgr.smithi097.swvonl (mgr.14180) 2166 : cluster [DBG] pgmap v1523: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:47.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:47 smithi193 bash[28559]: cluster 2023-11-17T07:36:46.082165+0000 mgr.smithi097.swvonl (mgr.14180) 2166 : cluster [DBG] pgmap v1523: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:49.175 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:49.176 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:49.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:49 smithi097 bash[24958]: cluster 2023-11-17T07:36:48.082694+0000 mgr.smithi097.swvonl (mgr.14180) 2167 : cluster [DBG] pgmap v1524: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:49.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:49 smithi097 bash[24958]: audit 2023-11-17T07:36:48.540003+0000 mon.smithi097 (mon.0) 1222 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.sdmmgy"}]: dispatch 2023-11-17T07:36:49.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:49 smithi193 bash[28559]: cluster 2023-11-17T07:36:48.082694+0000 mgr.smithi097.swvonl (mgr.14180) 2167 : cluster [DBG] pgmap v1524: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:36:49.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:49 smithi193 bash[28559]: audit 2023-11-17T07:36:48.540003+0000 mon.smithi097 (mon.0) 1222 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.sdmmgy"}]: dispatch 2023-11-17T07:36:50.193 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cephadm 2023-11-17T07:36:48.539187+0000 mgr.smithi097.swvonl (mgr.14180) 2168 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cephadm 2023-11-17T07:36:48.539519+0000 mgr.smithi097.swvonl (mgr.14180) 2169 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cephadm 2023-11-17T07:36:48.541015+0000 mgr.smithi097.swvonl (mgr.14180) 2170 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cephadm 2023-11-17T07:36:48.543086+0000 mgr.smithi097.swvonl (mgr.14180) 2171 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cephadm 2023-11-17T07:36:48.544869+0000 mgr.smithi097.swvonl (mgr.14180) 2172 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cluster 2023-11-17T07:36:48.545737+0000 mgr.smithi097.swvonl (mgr.14180) 2173 : cluster [DBG] pgmap v1525: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 233 B/s rd, 0 op/s 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: cluster 2023-11-17T07:36:49.136254+0000 mon.smithi097 (mon.0) 1223 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: audit 2023-11-17T07:36:49.168235+0000 mgr.smithi097.swvonl (mgr.14180) 2174 : audit [DBG] from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:50.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:50 smithi097 bash[24958]: audit 2023-11-17T07:36:50.108780+0000 mon.smithi097 (mon.0) 1224 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cephadm 2023-11-17T07:36:48.539187+0000 mgr.smithi097.swvonl (mgr.14180) 2168 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cephadm 2023-11-17T07:36:48.539519+0000 mgr.smithi097.swvonl (mgr.14180) 2169 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cephadm 2023-11-17T07:36:48.541015+0000 mgr.smithi097.swvonl (mgr.14180) 2170 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:36:50.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.sdmmgy ... 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cephadm 2023-11-17T07:36:48.543086+0000 mgr.smithi097.swvonl (mgr.14180) 2171 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cephadm 2023-11-17T07:36:48.544869+0000 mgr.smithi097.swvonl (mgr.14180) 2172 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cluster 2023-11-17T07:36:48.545737+0000 mgr.smithi097.swvonl (mgr.14180) 2173 : cluster [DBG] pgmap v1525: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 233 B/s rd, 0 op/s 2023-11-17T07:36:50.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: cluster 2023-11-17T07:36:49.136254+0000 mon.smithi097 (mon.0) 1223 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:36:50.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: audit 2023-11-17T07:36:49.168235+0000 mgr.smithi097.swvonl (mgr.14180) 2174 : audit [DBG] from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:50.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:50 smithi193 bash[28559]: audit 2023-11-17T07:36:50.108780+0000 mon.smithi097 (mon.0) 1224 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:51.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:52.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:52 smithi097 bash[24958]: cluster 2023-11-17T07:36:50.547181+0000 mgr.smithi097.swvonl (mgr.14180) 2175 : cluster [DBG] pgmap v1526: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:52.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:52 smithi193 bash[28559]: cluster 2023-11-17T07:36:50.547181+0000 mgr.smithi097.swvonl (mgr.14180) 2175 : cluster [DBG] pgmap v1526: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:54.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:54 smithi097 bash[24958]: cluster 2023-11-17T07:36:52.548075+0000 mgr.smithi097.swvonl (mgr.14180) 2176 : cluster [DBG] pgmap v1527: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:54.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:54 smithi193 bash[28559]: cluster 2023-11-17T07:36:52.548075+0000 mgr.smithi097.swvonl (mgr.14180) 2176 : cluster [DBG] pgmap v1527: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:54.973 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:36:54.974 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:36:55.959 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:36:56.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:56 smithi097 bash[24958]: cluster 2023-11-17T07:36:54.549222+0000 mgr.smithi097.swvonl (mgr.14180) 2177 : cluster [DBG] pgmap v1528: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:56.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:56 smithi097 bash[24958]: audit 2023-11-17T07:36:54.962383+0000 mgr.smithi097.swvonl (mgr.14180) 2178 : audit [DBG] from='client.15568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:56.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:56 smithi193 bash[28559]: cluster 2023-11-17T07:36:54.549222+0000 mgr.smithi097.swvonl (mgr.14180) 2177 : cluster [DBG] pgmap v1528: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:56.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:56 smithi193 bash[28559]: audit 2023-11-17T07:36:54.962383+0000 mgr.smithi097.swvonl (mgr.14180) 2178 : audit [DBG] from='client.15568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:36:56.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:36:58.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:57 smithi097 bash[24958]: cluster 2023-11-17T07:36:56.550594+0000 mgr.smithi097.swvonl (mgr.14180) 2179 : cluster [DBG] pgmap v1529: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:58.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:57 smithi097 bash[24958]: audit 2023-11-17T07:36:56.576257+0000 mon.smithi097 (mon.0) 1225 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:58.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:57 smithi193 bash[28559]: cluster 2023-11-17T07:36:56.550594+0000 mgr.smithi097.swvonl (mgr.14180) 2179 : cluster [DBG] pgmap v1529: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:36:58.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:57 smithi193 bash[28559]: audit 2023-11-17T07:36:56.576257+0000 mon.smithi097 (mon.0) 1225 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:36:59.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:36:59 smithi097 bash[24958]: cluster 2023-11-17T07:36:58.551575+0000 mgr.smithi097.swvonl (mgr.14180) 2180 : cluster [DBG] pgmap v1530: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:37:00.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:36:59 smithi193 bash[28559]: cluster 2023-11-17T07:36:58.551575+0000 mgr.smithi097.swvonl (mgr.14180) 2180 : cluster [DBG] pgmap v1530: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:37:00.833 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:00.833 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:01.754 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:01 smithi097 bash[24958]: cluster 2023-11-17T07:37:00.552938+0000 mgr.smithi097.swvonl (mgr.14180) 2181 : cluster [DBG] pgmap v1531: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:02.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:01 smithi097 bash[24958]: audit 2023-11-17T07:37:00.821329+0000 mgr.smithi097.swvonl (mgr.14180) 2182 : audit [DBG] from='client.15572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:02.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:01 smithi193 bash[28559]: cluster 2023-11-17T07:37:00.552938+0000 mgr.smithi097.swvonl (mgr.14180) 2181 : cluster [DBG] pgmap v1531: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:02.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:01 smithi193 bash[28559]: audit 2023-11-17T07:37:00.821329+0000 mgr.smithi097.swvonl (mgr.14180) 2182 : audit [DBG] from='client.15572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:02.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:04.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:04 smithi097 bash[24958]: cluster 2023-11-17T07:37:02.553912+0000 mgr.smithi097.swvonl (mgr.14180) 2183 : cluster [DBG] pgmap v1532: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:04.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:04 smithi193 bash[28559]: cluster 2023-11-17T07:37:02.553912+0000 mgr.smithi097.swvonl (mgr.14180) 2183 : cluster [DBG] pgmap v1532: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:06.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:06 smithi097 bash[24958]: cluster 2023-11-17T07:37:04.555058+0000 mgr.smithi097.swvonl (mgr.14180) 2184 : cluster [DBG] pgmap v1533: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:06.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:06 smithi193 bash[28559]: cluster 2023-11-17T07:37:04.555058+0000 mgr.smithi097.swvonl (mgr.14180) 2184 : cluster [DBG] pgmap v1533: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:06.568 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:06.568 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:07.523 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:08.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:08 smithi097 bash[24958]: cluster 2023-11-17T07:37:06.556403+0000 mgr.smithi097.swvonl (mgr.14180) 2185 : cluster [DBG] pgmap v1534: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:08.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:08 smithi097 bash[24958]: audit 2023-11-17T07:37:06.557449+0000 mgr.smithi097.swvonl (mgr.14180) 2186 : audit [DBG] from='client.15576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:08.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:08 smithi193 bash[28559]: cluster 2023-11-17T07:37:06.556403+0000 mgr.smithi097.swvonl (mgr.14180) 2185 : cluster [DBG] pgmap v1534: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:08.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:08 smithi193 bash[28559]: audit 2023-11-17T07:37:06.557449+0000 mgr.smithi097.swvonl (mgr.14180) 2186 : audit [DBG] from='client.15576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:08.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:10.086 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:10 smithi097 bash[24958]: cluster 2023-11-17T07:37:08.557511+0000 mgr.smithi097.swvonl (mgr.14180) 2187 : cluster [DBG] pgmap v1535: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:10.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:10 smithi193 bash[28559]: cluster 2023-11-17T07:37:08.557511+0000 mgr.smithi097.swvonl (mgr.14180) 2187 : cluster [DBG] pgmap v1535: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:12.300 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:12 smithi097 bash[24958]: cluster 2023-11-17T07:37:10.558959+0000 mgr.smithi097.swvonl (mgr.14180) 2188 : cluster [DBG] pgmap v1536: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:12.473 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:12.473 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:12.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:12 smithi193 bash[28559]: cluster 2023-11-17T07:37:10.558959+0000 mgr.smithi097.swvonl (mgr.14180) 2188 : cluster [DBG] pgmap v1536: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:13.434 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:14.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:14.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:14 smithi097 bash[24958]: audit 2023-11-17T07:37:12.462099+0000 mgr.smithi097.swvonl (mgr.14180) 2189 : audit [DBG] from='client.15580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:14.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:14 smithi097 bash[24958]: cluster 2023-11-17T07:37:12.559964+0000 mgr.smithi097.swvonl (mgr.14180) 2190 : cluster [DBG] pgmap v1537: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:14.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:14 smithi193 bash[28559]: audit 2023-11-17T07:37:12.462099+0000 mgr.smithi097.swvonl (mgr.14180) 2189 : audit [DBG] from='client.15580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:14.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:14 smithi193 bash[28559]: cluster 2023-11-17T07:37:12.559964+0000 mgr.smithi097.swvonl (mgr.14180) 2190 : cluster [DBG] pgmap v1537: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:16.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:16 smithi097 bash[24958]: cluster 2023-11-17T07:37:14.561047+0000 mgr.smithi097.swvonl (mgr.14180) 2191 : cluster [DBG] pgmap v1538: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:16.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:16 smithi193 bash[28559]: cluster 2023-11-17T07:37:14.561047+0000 mgr.smithi097.swvonl (mgr.14180) 2191 : cluster [DBG] pgmap v1538: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:18.203 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:18.204 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:18.508 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:18 smithi097 bash[24958]: cluster 2023-11-17T07:37:16.562475+0000 mgr.smithi097.swvonl (mgr.14180) 2192 : cluster [DBG] pgmap v1539: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:18.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:18 smithi193 bash[28559]: cluster 2023-11-17T07:37:16.562475+0000 mgr.smithi097.swvonl (mgr.14180) 2192 : cluster [DBG] pgmap v1539: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:19.205 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:19.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:19 smithi097 bash[24958]: audit 2023-11-17T07:37:18.191607+0000 mgr.smithi097.swvonl (mgr.14180) 2193 : audit [DBG] from='client.15584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:19 smithi193 bash[28559]: audit 2023-11-17T07:37:18.191607+0000 mgr.smithi097.swvonl (mgr.14180) 2193 : audit [DBG] from='client.15584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:20.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:20.400 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:20 smithi097 bash[24958]: cluster 2023-11-17T07:37:18.563554+0000 mgr.smithi097.swvonl (mgr.14180) 2194 : cluster [DBG] pgmap v1540: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:20.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:20 smithi193 bash[28559]: cluster 2023-11-17T07:37:18.563554+0000 mgr.smithi097.swvonl (mgr.14180) 2194 : cluster [DBG] pgmap v1540: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:22.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:22 smithi097 bash[24958]: cluster 2023-11-17T07:37:20.564398+0000 mgr.smithi097.swvonl (mgr.14180) 2195 : cluster [DBG] pgmap v1541: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:22.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:22 smithi193 bash[28559]: cluster 2023-11-17T07:37:20.564398+0000 mgr.smithi097.swvonl (mgr.14180) 2195 : cluster [DBG] pgmap v1541: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:24.173 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:32.450 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:32.452 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:24 smithi097 bash[24958]: cluster 2023-11-17T07:37:22.565329+0000 mgr.smithi097.swvonl (mgr.14180) 2196 : cluster [DBG] pgmap v1542: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.452 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:24 smithi193 bash[28559]: cluster 2023-11-17T07:37:22.565329+0000 mgr.smithi097.swvonl (mgr.14180) 2196 : cluster [DBG] pgmap v1542: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.452 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:25 smithi193 bash[28559]: audit 2023-11-17T07:37:24.160341+0000 mgr.smithi097.swvonl (mgr.14180) 2197 : audit [DBG] from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:32.453 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:26 smithi193 bash[28559]: cluster 2023-11-17T07:37:24.566491+0000 mgr.smithi097.swvonl (mgr.14180) 2198 : cluster [DBG] pgmap v1543: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.453 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:28 smithi193 bash[28559]: cluster 2023-11-17T07:37:26.567943+0000 mgr.smithi097.swvonl (mgr.14180) 2199 : cluster [DBG] pgmap v1544: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.453 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:30 smithi193 bash[28559]: cluster 2023-11-17T07:37:28.568974+0000 mgr.smithi097.swvonl (mgr.14180) 2200 : cluster [DBG] pgmap v1545: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.453 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:32.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:25 smithi097 bash[24958]: audit 2023-11-17T07:37:24.160341+0000 mgr.smithi097.swvonl (mgr.14180) 2197 : audit [DBG] from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:32.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:26 smithi097 bash[24958]: cluster 2023-11-17T07:37:24.566491+0000 mgr.smithi097.swvonl (mgr.14180) 2198 : cluster [DBG] pgmap v1543: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:28 smithi097 bash[24958]: cluster 2023-11-17T07:37:26.567943+0000 mgr.smithi097.swvonl (mgr.14180) 2199 : cluster [DBG] pgmap v1544: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.454 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:30 smithi097 bash[24958]: cluster 2023-11-17T07:37:28.568974+0000 mgr.smithi097.swvonl (mgr.14180) 2200 : cluster [DBG] pgmap v1545: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:32.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:32 smithi097 bash[24958]: cluster 2023-11-17T07:37:30.570453+0000 mgr.smithi097.swvonl (mgr.14180) 2201 : cluster [DBG] pgmap v1546: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:32.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:32 smithi193 bash[28559]: cluster 2023-11-17T07:37:30.570453+0000 mgr.smithi097.swvonl (mgr.14180) 2201 : cluster [DBG] pgmap v1546: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:33.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:34.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:34 smithi097 bash[24958]: cluster 2023-11-17T07:37:32.571405+0000 mgr.smithi097.swvonl (mgr.14180) 2202 : cluster [DBG] pgmap v1547: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:34.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:34 smithi193 bash[28559]: cluster 2023-11-17T07:37:32.571405+0000 mgr.smithi097.swvonl (mgr.14180) 2202 : cluster [DBG] pgmap v1547: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:36.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:36 smithi097 bash[24958]: cluster 2023-11-17T07:37:34.572446+0000 mgr.smithi097.swvonl (mgr.14180) 2203 : cluster [DBG] pgmap v1548: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:36.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:36 smithi193 bash[28559]: cluster 2023-11-17T07:37:34.572446+0000 mgr.smithi097.swvonl (mgr.14180) 2203 : cluster [DBG] pgmap v1548: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:37.389 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:37.389 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:38.327 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:38 smithi097 bash[24958]: cluster 2023-11-17T07:37:36.573806+0000 mgr.smithi097.swvonl (mgr.14180) 2204 : cluster [DBG] pgmap v1549: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:38.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:38 smithi193 bash[28559]: cluster 2023-11-17T07:37:36.573806+0000 mgr.smithi097.swvonl (mgr.14180) 2204 : cluster [DBG] pgmap v1549: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:39.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:39.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:39 smithi097 bash[24958]: audit 2023-11-17T07:37:37.376892+0000 mgr.smithi097.swvonl (mgr.14180) 2205 : audit [DBG] from='client.15592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:39 smithi193 bash[28559]: audit 2023-11-17T07:37:37.376892+0000 mgr.smithi097.swvonl (mgr.14180) 2205 : audit [DBG] from='client.15592 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:40.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:40 smithi097 bash[24958]: cluster 2023-11-17T07:37:38.574964+0000 mgr.smithi097.swvonl (mgr.14180) 2206 : cluster [DBG] pgmap v1550: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:40.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:40 smithi193 bash[28559]: cluster 2023-11-17T07:37:38.574964+0000 mgr.smithi097.swvonl (mgr.14180) 2206 : cluster [DBG] pgmap v1550: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:42.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:42 smithi097 bash[24958]: cluster 2023-11-17T07:37:40.576360+0000 mgr.smithi097.swvonl (mgr.14180) 2207 : cluster [DBG] pgmap v1551: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:42.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:42 smithi193 bash[28559]: cluster 2023-11-17T07:37:40.576360+0000 mgr.smithi097.swvonl (mgr.14180) 2207 : cluster [DBG] pgmap v1551: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:37:43.277 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:37:43.278 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:37:44.176 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:37:44.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:44 smithi097 bash[24958]: cluster 2023-11-17T07:37:42.577431+0000 mgr.smithi097.swvonl (mgr.14180) 2208 : cluster [DBG] pgmap v1552: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:44.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:44 smithi193 bash[28559]: cluster 2023-11-17T07:37:42.577431+0000 mgr.smithi097.swvonl (mgr.14180) 2208 : cluster [DBG] pgmap v1552: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:45.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:37:45.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:45 smithi097 bash[24958]: audit 2023-11-17T07:37:43.265054+0000 mgr.smithi097.swvonl (mgr.14180) 2209 : audit [DBG] from='client.15596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:45.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:45 smithi193 bash[28559]: audit 2023-11-17T07:37:43.265054+0000 mgr.smithi097.swvonl (mgr.14180) 2209 : audit [DBG] from='client.15596 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:37:46.507 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:46 smithi097 bash[24958]: cluster 2023-11-17T07:37:44.578511+0000 mgr.smithi097.swvonl (mgr.14180) 2210 : cluster [DBG] pgmap v1553: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:46.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:46 smithi193 bash[28559]: cluster 2023-11-17T07:37:44.578511+0000 mgr.smithi097.swvonl (mgr.14180) 2210 : cluster [DBG] pgmap v1553: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:37:48.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:48 smithi097 bash[24958]: cluster 2023-11-17T07:37:46.579958+0000 mgr.smithi097.swvonl (mgr.14180) 2211 : cluster [DBG] pgmap v1554: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:09.090 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:38:09.090 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:35:39.756524Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.ekpcvp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:44.301811Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.kzeexl on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-kzeexl\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.kzeexl\nDeploy daemon haproxy.nfs.foo.smithi193.kzeexl ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.540898Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.sdmmgy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-sdmmgy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.sdmmgy\nDeploy daemon haproxy.nfs.foo.smithi097.sdmmgy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:36:48.542974Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.mgpftx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.548245+0000 mon.smithi097 (mon.0) 1226 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.854576+0000 mon.smithi097 (mon.0) 1227 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.855866+0000 mon.smithi097 (mon.0) 1228 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.863275+0000 mon.smithi097 (mon.0) 1229 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.872355+0000 mon.smithi097 (mon.0) 1230 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:38:09.091 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:49 smithi097 bash[24958]: audit 2023-11-17T07:37:48.886858+0000 mon.smithi097 (mon.0) 1231 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.093 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:48 smithi193 bash[28559]: cluster 2023-11-17T07:37:46.579958+0000 mgr.smithi097.swvonl (mgr.14180) 2211 : cluster [DBG] pgmap v1554: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.548245+0000 mon.smithi097 (mon.0) 1226 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.854576+0000 mon.smithi097 (mon.0) 1227 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.855866+0000 mon.smithi097 (mon.0) 1228 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.863275+0000 mon.smithi097 (mon.0) 1229 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.872355+0000 mon.smithi097 (mon.0) 1230 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:49 smithi193 bash[28559]: audit 2023-11-17T07:37:48.886858+0000 mon.smithi097 (mon.0) 1231 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.095 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: cluster 2023-11-17T07:37:48.581119+0000 mgr.smithi097.swvonl (mgr.14180) 2212 : cluster [DBG] pgmap v1555: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: cluster 2023-11-17T07:37:48.864187+0000 mgr.smithi097.swvonl (mgr.14180) 2213 : cluster [DBG] pgmap v1556: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: cephadm 2023-11-17T07:37:48.890944+0000 mgr.smithi097.swvonl (mgr.14180) 2214 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rlkuvv on smithi193 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: audit 2023-11-17T07:37:48.991149+0000 mgr.smithi097.swvonl (mgr.14180) 2215 : audit [DBG] from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: cluster 2023-11-17T07:37:49.186265+0000 mon.smithi097 (mon.0) 1232 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:50 smithi193 bash[28559]: cluster 2023-11-17T07:37:49.186294+0000 mon.smithi097 (mon.0) 1233 : cluster [INF] Cluster is now healthy 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:52 smithi193 bash[28559]: cluster 2023-11-17T07:37:50.865323+0000 mgr.smithi097.swvonl (mgr.14180) 2216 : cluster [DBG] pgmap v1557: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:53 smithi193 bash[28559]: audit 2023-11-17T07:37:52.671097+0000 mon.smithi097 (mon.0) 1234 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rlkuvv"}]: dispatch 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: cephadm 2023-11-17T07:37:52.669830+0000 mgr.smithi097.swvonl (mgr.14180) 2217 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.096 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.097 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: cephadm 2023-11-17T07:37:52.670467+0000 mgr.smithi097.swvonl (mgr.14180) 2218 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.098 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: cephadm 2023-11-17T07:37:52.672585+0000 mgr.smithi097.swvonl (mgr.14180) 2219 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rlkuvv on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: cephadm 2023-11-17T07:37:52.677679+0000 mgr.smithi097.swvonl (mgr.14180) 2220 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ngiufh on smithi097 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:54 smithi193 bash[28559]: cluster 2023-11-17T07:37:52.866079+0000 mgr.smithi097.swvonl (mgr.14180) 2221 : cluster [DBG] pgmap v1558: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.099 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:55 smithi193 bash[28559]: cluster 2023-11-17T07:37:54.867195+0000 mgr.smithi097.swvonl (mgr.14180) 2222 : cluster [DBG] pgmap v1559: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:56 smithi193 bash[28559]: audit 2023-11-17T07:37:56.432148+0000 mon.smithi097 (mon.0) 1235 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ngiufh"}]: dispatch 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cephadm 2023-11-17T07:37:56.431079+0000 mgr.smithi097.swvonl (mgr.14180) 2223 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.100 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.101 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cephadm 2023-11-17T07:37:56.431591+0000 mgr.smithi097.swvonl (mgr.14180) 2224 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cephadm 2023-11-17T07:37:56.433418+0000 mgr.smithi097.swvonl (mgr.14180) 2225 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ngiufh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.102 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cephadm 2023-11-17T07:37:56.437450+0000 mgr.smithi097.swvonl (mgr.14180) 2226 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vjrimr on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cephadm 2023-11-17T07:37:56.440611+0000 mgr.smithi097.swvonl (mgr.14180) 2227 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.blimzu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cluster 2023-11-17T07:37:56.441863+0000 mgr.smithi097.swvonl (mgr.14180) 2228 : cluster [DBG] pgmap v1560: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:57 smithi193 bash[28559]: cluster 2023-11-17T07:37:56.493361+0000 mon.smithi097 (mon.0) 1236 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:37:59 smithi193 bash[28559]: cluster 2023-11-17T07:37:58.442922+0000 mgr.smithi097.swvonl (mgr.14180) 2229 : cluster [DBG] pgmap v1561: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:01 smithi193 bash[28559]: audit 2023-11-17T07:38:00.158660+0000 mon.smithi097 (mon.0) 1237 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:02 smithi193 bash[28559]: cluster 2023-11-17T07:38:00.444304+0000 mgr.smithi097.swvonl (mgr.14180) 2230 : cluster [DBG] pgmap v1562: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:38:09.103 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:04 smithi193 bash[28559]: cluster 2023-11-17T07:38:02.445232+0000 mgr.smithi097.swvonl (mgr.14180) 2231 : cluster [DBG] pgmap v1563: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:09.104 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:06 smithi193 bash[28559]: cluster 2023-11-17T07:38:04.446050+0000 mgr.smithi097.swvonl (mgr.14180) 2232 : cluster [DBG] pgmap v1564: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:09.104 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:08 smithi193 bash[28559]: cluster 2023-11-17T07:38:06.447316+0000 mgr.smithi097.swvonl (mgr.14180) 2233 : cluster [DBG] pgmap v1565: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:09.104 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: cluster 2023-11-17T07:37:48.581119+0000 mgr.smithi097.swvonl (mgr.14180) 2212 : cluster [DBG] pgmap v1555: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: cluster 2023-11-17T07:37:48.864187+0000 mgr.smithi097.swvonl (mgr.14180) 2213 : cluster [DBG] pgmap v1556: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: cephadm 2023-11-17T07:37:48.890944+0000 mgr.smithi097.swvonl (mgr.14180) 2214 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rlkuvv on smithi193 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: audit 2023-11-17T07:37:48.991149+0000 mgr.smithi097.swvonl (mgr.14180) 2215 : audit [DBG] from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: cluster 2023-11-17T07:37:49.186265+0000 mon.smithi097 (mon.0) 1232 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:50 smithi097 bash[24958]: cluster 2023-11-17T07:37:49.186294+0000 mon.smithi097 (mon.0) 1233 : cluster [INF] Cluster is now healthy 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:52 smithi097 bash[24958]: cluster 2023-11-17T07:37:50.865323+0000 mgr.smithi097.swvonl (mgr.14180) 2216 : cluster [DBG] pgmap v1557: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:53 smithi097 bash[24958]: audit 2023-11-17T07:37:52.671097+0000 mon.smithi097 (mon.0) 1234 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rlkuvv"}]: dispatch 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: cephadm 2023-11-17T07:37:52.669830+0000 mgr.smithi097.swvonl (mgr.14180) 2217 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.105 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:38:09.106 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: cephadm 2023-11-17T07:37:52.670467+0000 mgr.smithi097.swvonl (mgr.14180) 2218 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.107 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: cephadm 2023-11-17T07:37:52.672585+0000 mgr.smithi097.swvonl (mgr.14180) 2219 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rlkuvv on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rlkuvv ... 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: cephadm 2023-11-17T07:37:52.677679+0000 mgr.smithi097.swvonl (mgr.14180) 2220 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ngiufh on smithi097 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:54 smithi097 bash[24958]: cluster 2023-11-17T07:37:52.866079+0000 mgr.smithi097.swvonl (mgr.14180) 2221 : cluster [DBG] pgmap v1558: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.108 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:55 smithi097 bash[24958]: cluster 2023-11-17T07:37:54.867195+0000 mgr.smithi097.swvonl (mgr.14180) 2222 : cluster [DBG] pgmap v1559: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:56 smithi097 bash[24958]: audit 2023-11-17T07:37:56.432148+0000 mon.smithi097 (mon.0) 1235 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ngiufh"}]: dispatch 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cephadm 2023-11-17T07:37:56.431079+0000 mgr.smithi097.swvonl (mgr.14180) 2223 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.109 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.110 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cephadm 2023-11-17T07:37:56.431591+0000 mgr.smithi097.swvonl (mgr.14180) 2224 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cephadm 2023-11-17T07:37:56.433418+0000 mgr.smithi097.swvonl (mgr.14180) 2225 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ngiufh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.111 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ngiufh ... 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cephadm 2023-11-17T07:37:56.437450+0000 mgr.smithi097.swvonl (mgr.14180) 2226 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.vjrimr on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cephadm 2023-11-17T07:37:56.440611+0000 mgr.smithi097.swvonl (mgr.14180) 2227 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.blimzu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cluster 2023-11-17T07:37:56.441863+0000 mgr.smithi097.swvonl (mgr.14180) 2228 : cluster [DBG] pgmap v1560: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:57 smithi097 bash[24958]: cluster 2023-11-17T07:37:56.493361+0000 mon.smithi097 (mon.0) 1236 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:37:59 smithi097 bash[24958]: cluster 2023-11-17T07:37:58.442922+0000 mgr.smithi097.swvonl (mgr.14180) 2229 : cluster [DBG] pgmap v1561: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 207 B/s rd, 0 op/s 2023-11-17T07:38:09.112 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:01 smithi097 bash[24958]: audit 2023-11-17T07:38:00.158660+0000 mon.smithi097 (mon.0) 1237 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:38:09.113 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:02 smithi097 bash[24958]: cluster 2023-11-17T07:38:00.444304+0000 mgr.smithi097.swvonl (mgr.14180) 2230 : cluster [DBG] pgmap v1562: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 353 B/s rd, 0 op/s 2023-11-17T07:38:09.113 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:04 smithi097 bash[24958]: cluster 2023-11-17T07:38:02.445232+0000 mgr.smithi097.swvonl (mgr.14180) 2231 : cluster [DBG] pgmap v1563: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:09.113 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:06 smithi097 bash[24958]: cluster 2023-11-17T07:38:04.446050+0000 mgr.smithi097.swvonl (mgr.14180) 2232 : cluster [DBG] pgmap v1564: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:09.113 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:08 smithi097 bash[24958]: cluster 2023-11-17T07:38:06.447316+0000 mgr.smithi097.swvonl (mgr.14180) 2233 : cluster [DBG] pgmap v1565: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 176 B/s rd, 0 op/s 2023-11-17T07:38:10.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:38:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:10 smithi097 bash[24958]: cluster 2023-11-17T07:38:08.448447+0000 mgr.smithi097.swvonl (mgr.14180) 2234 : cluster [DBG] pgmap v1566: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:10 smithi193 bash[28559]: cluster 2023-11-17T07:38:08.448447+0000 mgr.smithi097.swvonl (mgr.14180) 2234 : cluster [DBG] pgmap v1566: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:38:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:12 smithi097 bash[24958]: cluster 2023-11-17T07:38:10.449385+0000 mgr.smithi097.swvonl (mgr.14180) 2235 : cluster [DBG] pgmap v1567: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:38:12.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:12 smithi193 bash[28559]: cluster 2023-11-17T07:38:10.449385+0000 mgr.smithi097.swvonl (mgr.14180) 2235 : cluster [DBG] pgmap v1567: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:38:14.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:05.426 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:36:48.544772Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.gdfzmp on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:37:52.672364Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rlkuvv on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rlkuvv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rlkuvv\nDeploy daemon haproxy.nfs.foo.smithi193.rlkuvv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:37:56.433192Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ngiufh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ngiufh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ngiufh\nDeploy daemon haproxy.nfs.foo.smithi097.ngiufh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:37:56.437267Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.vjrimr on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:37:56.440441Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.blimzu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:05.430 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:14 smithi097 bash[24958]: cluster 2023-11-17T07:38:12.450318+0000 mgr.smithi097.swvonl (mgr.14180) 2236 : cluster [DBG] pgmap v1568: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.433 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:14 smithi193 bash[28559]: cluster 2023-11-17T07:38:12.450318+0000 mgr.smithi097.swvonl (mgr.14180) 2236 : cluster [DBG] pgmap v1568: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:15 smithi193 bash[28559]: audit 2023-11-17T07:38:14.095492+0000 mgr.smithi097.swvonl (mgr.14180) 2237 : audit [DBG] from='client.15604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:16 smithi193 bash[28559]: cluster 2023-11-17T07:38:14.450952+0000 mgr.smithi097.swvonl (mgr.14180) 2238 : cluster [DBG] pgmap v1569: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:18 smithi193 bash[28559]: cluster 2023-11-17T07:38:16.452280+0000 mgr.smithi097.swvonl (mgr.14180) 2239 : cluster [DBG] pgmap v1570: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:20 smithi193 bash[28559]: cluster 2023-11-17T07:38:18.453315+0000 mgr.smithi097.swvonl (mgr.14180) 2240 : cluster [DBG] pgmap v1571: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.465 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:22 smithi193 bash[28559]: cluster 2023-11-17T07:38:20.454712+0000 mgr.smithi097.swvonl (mgr.14180) 2241 : cluster [DBG] pgmap v1572: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.466 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:24 smithi193 bash[28559]: cluster 2023-11-17T07:38:22.455754+0000 mgr.smithi097.swvonl (mgr.14180) 2242 : cluster [DBG] pgmap v1573: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.466 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:26 smithi193 bash[28559]: cluster 2023-11-17T07:38:24.456959+0000 mgr.smithi097.swvonl (mgr.14180) 2243 : cluster [DBG] pgmap v1574: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.466 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:28 smithi193 bash[28559]: cluster 2023-11-17T07:38:26.458361+0000 mgr.smithi097.swvonl (mgr.14180) 2244 : cluster [DBG] pgmap v1575: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.466 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:30 smithi193 bash[28559]: cluster 2023-11-17T07:38:28.459391+0000 mgr.smithi097.swvonl (mgr.14180) 2245 : cluster [DBG] pgmap v1576: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.466 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:32 smithi193 bash[28559]: cluster 2023-11-17T07:38:30.460842+0000 mgr.smithi097.swvonl (mgr.14180) 2246 : cluster [DBG] pgmap v1577: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:34 smithi193 bash[28559]: cluster 2023-11-17T07:38:32.461745+0000 mgr.smithi097.swvonl (mgr.14180) 2247 : cluster [DBG] pgmap v1578: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:36 smithi193 bash[28559]: cluster 2023-11-17T07:38:34.462732+0000 mgr.smithi097.swvonl (mgr.14180) 2248 : cluster [DBG] pgmap v1579: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:38 smithi193 bash[28559]: cluster 2023-11-17T07:38:36.464188+0000 mgr.smithi097.swvonl (mgr.14180) 2249 : cluster [DBG] pgmap v1580: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.474 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:40 smithi193 bash[28559]: cluster 2023-11-17T07:38:38.465255+0000 mgr.smithi097.swvonl (mgr.14180) 2250 : cluster [DBG] pgmap v1581: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:42 smithi193 bash[28559]: cluster 2023-11-17T07:38:40.466701+0000 mgr.smithi097.swvonl (mgr.14180) 2251 : cluster [DBG] pgmap v1582: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:44 smithi193 bash[28559]: cluster 2023-11-17T07:38:42.467808+0000 mgr.smithi097.swvonl (mgr.14180) 2252 : cluster [DBG] pgmap v1583: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:46 smithi193 bash[28559]: cluster 2023-11-17T07:38:44.468811+0000 mgr.smithi097.swvonl (mgr.14180) 2253 : cluster [DBG] pgmap v1584: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:48 smithi193 bash[28559]: cluster 2023-11-17T07:38:46.470091+0000 mgr.smithi097.swvonl (mgr.14180) 2254 : cluster [DBG] pgmap v1585: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:50 smithi193 bash[28559]: cluster 2023-11-17T07:38:48.471079+0000 mgr.smithi097.swvonl (mgr.14180) 2255 : cluster [DBG] pgmap v1586: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:52 smithi193 bash[28559]: cluster 2023-11-17T07:38:50.472455+0000 mgr.smithi097.swvonl (mgr.14180) 2256 : cluster [DBG] pgmap v1587: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:54 smithi193 bash[28559]: cluster 2023-11-17T07:38:52.473345+0000 mgr.smithi097.swvonl (mgr.14180) 2257 : cluster [DBG] pgmap v1588: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:56 smithi193 bash[28559]: cluster 2023-11-17T07:38:54.474434+0000 mgr.smithi097.swvonl (mgr.14180) 2258 : cluster [DBG] pgmap v1589: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.476 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:56.445571+0000 mon.smithi097 (mon.0) 1238 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:56.716868+0000 mon.smithi097 (mon.0) 1239 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:56.990796+0000 mon.smithi097 (mon.0) 1240 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:56.992178+0000 mon.smithi097 (mon.0) 1241 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:56.998734+0000 mon.smithi097 (mon.0) 1242 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:57.005312+0000 mon.smithi097 (mon.0) 1243 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:57 smithi193 bash[28559]: audit 2023-11-17T07:38:57.018280+0000 mon.smithi097 (mon.0) 1244 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:58 smithi193 bash[28559]: cluster 2023-11-17T07:38:56.475651+0000 mgr.smithi097.swvonl (mgr.14180) 2259 : cluster [DBG] pgmap v1590: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:58 smithi193 bash[28559]: cluster 2023-11-17T07:38:56.999539+0000 mgr.smithi097.swvonl (mgr.14180) 2260 : cluster [DBG] pgmap v1591: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:58 smithi193 bash[28559]: cephadm 2023-11-17T07:38:57.023167+0000 mgr.smithi097.swvonl (mgr.14180) 2261 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ymtnkn on smithi193 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:58 smithi193 bash[28559]: cluster 2023-11-17T07:38:57.163516+0000 mon.smithi097 (mon.0) 1245 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:58 smithi193 bash[28559]: cluster 2023-11-17T07:38:57.163599+0000 mon.smithi097 (mon.0) 1246 : cluster [INF] Cluster is now healthy 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:38:59 smithi193 bash[28559]: cluster 2023-11-17T07:38:59.000416+0000 mgr.smithi097.swvonl (mgr.14180) 2262 : cluster [DBG] pgmap v1592: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.477 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: cluster 2023-11-17T07:39:01.001569+0000 mgr.smithi097.swvonl (mgr.14180) 2263 : cluster [DBG] pgmap v1593: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: cephadm 2023-11-17T07:39:01.124321+0000 mgr.smithi097.swvonl (mgr.14180) 2264 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.478 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:05.487 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.488 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: cephadm 2023-11-17T07:39:01.124918+0000 mgr.smithi097.swvonl (mgr.14180) 2265 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: audit 2023-11-17T07:39:01.125524+0000 mon.smithi097 (mon.0) 1247 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ymtnkn"}]: dispatch 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: cephadm 2023-11-17T07:39:01.127323+0000 mgr.smithi097.swvonl (mgr.14180) 2266 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ymtnkn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.489 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:02 smithi193 bash[28559]: cephadm 2023-11-17T07:39:01.130984+0000 mgr.smithi097.swvonl (mgr.14180) 2267 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pqfllk on smithi097 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:04 smithi193 bash[28559]: cluster 2023-11-17T07:39:03.002325+0000 mgr.smithi097.swvonl (mgr.14180) 2268 : cluster [DBG] pgmap v1594: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:05 smithi193 bash[28559]: audit 2023-11-17T07:39:04.955212+0000 mon.smithi097 (mon.0) 1248 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pqfllk"}]: dispatch 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cephadm 2023-11-17T07:39:04.953889+0000 mgr.smithi097.swvonl (mgr.14180) 2269 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.490 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:40:05.491 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.492 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cephadm 2023-11-17T07:39:04.954424+0000 mgr.smithi097.swvonl (mgr.14180) 2270 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cephadm 2023-11-17T07:39:04.956846+0000 mgr.smithi097.swvonl (mgr.14180) 2271 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pqfllk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:05.493 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.494 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cephadm 2023-11-17T07:39:04.961751+0000 mgr.smithi097.swvonl (mgr.14180) 2272 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.nflyro on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cephadm 2023-11-17T07:39:04.963518+0000 mgr.smithi097.swvonl (mgr.14180) 2273 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cluster 2023-11-17T07:39:04.964743+0000 mgr.smithi097.swvonl (mgr.14180) 2274 : cluster [DBG] pgmap v1595: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: cluster 2023-11-17T07:39:05.077474+0000 mon.smithi097 (mon.0) 1249 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:06 smithi193 bash[28559]: audit 2023-11-17T07:39:05.209800+0000 mon.smithi097 (mon.0) 1250 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:08 smithi193 bash[28559]: cluster 2023-11-17T07:39:06.965717+0000 mgr.smithi097.swvonl (mgr.14180) 2275 : cluster [DBG] pgmap v1596: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:40:05.495 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:10 smithi193 bash[28559]: cluster 2023-11-17T07:39:08.966837+0000 mgr.smithi097.swvonl (mgr.14180) 2276 : cluster [DBG] pgmap v1597: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:12 smithi193 bash[28559]: cluster 2023-11-17T07:39:10.968187+0000 mgr.smithi097.swvonl (mgr.14180) 2277 : cluster [DBG] pgmap v1598: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 342 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:14 smithi193 bash[28559]: cluster 2023-11-17T07:39:12.969134+0000 mgr.smithi097.swvonl (mgr.14180) 2278 : cluster [DBG] pgmap v1599: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:16 smithi193 bash[28559]: cluster 2023-11-17T07:39:14.970521+0000 mgr.smithi097.swvonl (mgr.14180) 2279 : cluster [DBG] pgmap v1600: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:18 smithi193 bash[28559]: cluster 2023-11-17T07:39:16.971878+0000 mgr.smithi097.swvonl (mgr.14180) 2280 : cluster [DBG] pgmap v1601: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:20 smithi193 bash[28559]: cluster 2023-11-17T07:39:18.972958+0000 mgr.smithi097.swvonl (mgr.14180) 2281 : cluster [DBG] pgmap v1602: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:22 smithi193 bash[28559]: cluster 2023-11-17T07:39:20.974262+0000 mgr.smithi097.swvonl (mgr.14180) 2282 : cluster [DBG] pgmap v1603: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.496 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:24 smithi193 bash[28559]: cluster 2023-11-17T07:39:22.975243+0000 mgr.smithi097.swvonl (mgr.14180) 2283 : cluster [DBG] pgmap v1604: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:26 smithi193 bash[28559]: cluster 2023-11-17T07:39:24.975951+0000 mgr.smithi097.swvonl (mgr.14180) 2284 : cluster [DBG] pgmap v1605: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:28 smithi193 bash[28559]: cluster 2023-11-17T07:39:26.976922+0000 mgr.smithi097.swvonl (mgr.14180) 2285 : cluster [DBG] pgmap v1606: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:30 smithi193 bash[28559]: cluster 2023-11-17T07:39:28.978062+0000 mgr.smithi097.swvonl (mgr.14180) 2286 : cluster [DBG] pgmap v1607: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:32 smithi193 bash[28559]: cluster 2023-11-17T07:39:30.979497+0000 mgr.smithi097.swvonl (mgr.14180) 2287 : cluster [DBG] pgmap v1608: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:34 smithi193 bash[28559]: cluster 2023-11-17T07:39:32.980256+0000 mgr.smithi097.swvonl (mgr.14180) 2288 : cluster [DBG] pgmap v1609: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.497 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:36 smithi193 bash[28559]: cluster 2023-11-17T07:39:34.981540+0000 mgr.smithi097.swvonl (mgr.14180) 2289 : cluster [DBG] pgmap v1610: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:38 smithi193 bash[28559]: cluster 2023-11-17T07:39:36.982621+0000 mgr.smithi097.swvonl (mgr.14180) 2290 : cluster [DBG] pgmap v1611: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:40 smithi193 bash[28559]: cluster 2023-11-17T07:39:38.983645+0000 mgr.smithi097.swvonl (mgr.14180) 2291 : cluster [DBG] pgmap v1612: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:42 smithi193 bash[28559]: cluster 2023-11-17T07:39:40.984971+0000 mgr.smithi097.swvonl (mgr.14180) 2292 : cluster [DBG] pgmap v1613: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:44 smithi193 bash[28559]: cluster 2023-11-17T07:39:42.985941+0000 mgr.smithi097.swvonl (mgr.14180) 2293 : cluster [DBG] pgmap v1614: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:46 smithi193 bash[28559]: cluster 2023-11-17T07:39:44.987329+0000 mgr.smithi097.swvonl (mgr.14180) 2294 : cluster [DBG] pgmap v1615: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.498 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:48 smithi193 bash[28559]: cluster 2023-11-17T07:39:46.988371+0000 mgr.smithi097.swvonl (mgr.14180) 2295 : cluster [DBG] pgmap v1616: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:50 smithi193 bash[28559]: cluster 2023-11-17T07:39:48.989229+0000 mgr.smithi097.swvonl (mgr.14180) 2296 : cluster [DBG] pgmap v1617: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:52 smithi193 bash[28559]: cluster 2023-11-17T07:39:50.990605+0000 mgr.smithi097.swvonl (mgr.14180) 2297 : cluster [DBG] pgmap v1618: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:53 smithi193 bash[28559]: cluster 2023-11-17T07:39:52.991557+0000 mgr.smithi097.swvonl (mgr.14180) 2298 : cluster [DBG] pgmap v1619: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:56 smithi193 bash[28559]: cluster 2023-11-17T07:39:54.992904+0000 mgr.smithi097.swvonl (mgr.14180) 2299 : cluster [DBG] pgmap v1620: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:39:58 smithi193 bash[28559]: cluster 2023-11-17T07:39:56.993523+0000 mgr.smithi097.swvonl (mgr.14180) 2300 : cluster [DBG] pgmap v1621: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:00 smithi193 bash[28559]: cluster 2023-11-17T07:39:58.994644+0000 mgr.smithi097.swvonl (mgr.14180) 2301 : cluster [DBG] pgmap v1622: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.499 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:00 smithi193 bash[28559]: cluster 2023-11-17T07:40:00.000171+0000 mon.smithi097 (mon.0) 1251 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:40:05.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:02 smithi193 bash[28559]: cluster 2023-11-17T07:40:00.996081+0000 mgr.smithi097.swvonl (mgr.14180) 2302 : cluster [DBG] pgmap v1623: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.500 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:04 smithi193 bash[28559]: cluster 2023-11-17T07:40:02.997237+0000 mgr.smithi097.swvonl (mgr.14180) 2303 : cluster [DBG] pgmap v1624: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.501 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:15 smithi097 bash[24958]: audit 2023-11-17T07:38:14.095492+0000 mgr.smithi097.swvonl (mgr.14180) 2237 : audit [DBG] from='client.15604 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:05.501 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:16 smithi097 bash[24958]: cluster 2023-11-17T07:38:14.450952+0000 mgr.smithi097.swvonl (mgr.14180) 2238 : cluster [DBG] pgmap v1569: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:18 smithi097 bash[24958]: cluster 2023-11-17T07:38:16.452280+0000 mgr.smithi097.swvonl (mgr.14180) 2239 : cluster [DBG] pgmap v1570: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:20 smithi097 bash[24958]: cluster 2023-11-17T07:38:18.453315+0000 mgr.smithi097.swvonl (mgr.14180) 2240 : cluster [DBG] pgmap v1571: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:22 smithi097 bash[24958]: cluster 2023-11-17T07:38:20.454712+0000 mgr.smithi097.swvonl (mgr.14180) 2241 : cluster [DBG] pgmap v1572: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:24 smithi097 bash[24958]: cluster 2023-11-17T07:38:22.455754+0000 mgr.smithi097.swvonl (mgr.14180) 2242 : cluster [DBG] pgmap v1573: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.502 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:26 smithi097 bash[24958]: cluster 2023-11-17T07:38:24.456959+0000 mgr.smithi097.swvonl (mgr.14180) 2243 : cluster [DBG] pgmap v1574: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:28 smithi097 bash[24958]: cluster 2023-11-17T07:38:26.458361+0000 mgr.smithi097.swvonl (mgr.14180) 2244 : cluster [DBG] pgmap v1575: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:30 smithi097 bash[24958]: cluster 2023-11-17T07:38:28.459391+0000 mgr.smithi097.swvonl (mgr.14180) 2245 : cluster [DBG] pgmap v1576: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:32 smithi097 bash[24958]: cluster 2023-11-17T07:38:30.460842+0000 mgr.smithi097.swvonl (mgr.14180) 2246 : cluster [DBG] pgmap v1577: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:34 smithi097 bash[24958]: cluster 2023-11-17T07:38:32.461745+0000 mgr.smithi097.swvonl (mgr.14180) 2247 : cluster [DBG] pgmap v1578: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:36 smithi097 bash[24958]: cluster 2023-11-17T07:38:34.462732+0000 mgr.smithi097.swvonl (mgr.14180) 2248 : cluster [DBG] pgmap v1579: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.503 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:38 smithi097 bash[24958]: cluster 2023-11-17T07:38:36.464188+0000 mgr.smithi097.swvonl (mgr.14180) 2249 : cluster [DBG] pgmap v1580: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:40 smithi097 bash[24958]: cluster 2023-11-17T07:38:38.465255+0000 mgr.smithi097.swvonl (mgr.14180) 2250 : cluster [DBG] pgmap v1581: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:42 smithi097 bash[24958]: cluster 2023-11-17T07:38:40.466701+0000 mgr.smithi097.swvonl (mgr.14180) 2251 : cluster [DBG] pgmap v1582: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:44 smithi097 bash[24958]: cluster 2023-11-17T07:38:42.467808+0000 mgr.smithi097.swvonl (mgr.14180) 2252 : cluster [DBG] pgmap v1583: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:46 smithi097 bash[24958]: cluster 2023-11-17T07:38:44.468811+0000 mgr.smithi097.swvonl (mgr.14180) 2253 : cluster [DBG] pgmap v1584: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:48 smithi097 bash[24958]: cluster 2023-11-17T07:38:46.470091+0000 mgr.smithi097.swvonl (mgr.14180) 2254 : cluster [DBG] pgmap v1585: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:50 smithi097 bash[24958]: cluster 2023-11-17T07:38:48.471079+0000 mgr.smithi097.swvonl (mgr.14180) 2255 : cluster [DBG] pgmap v1586: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.504 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:52 smithi097 bash[24958]: cluster 2023-11-17T07:38:50.472455+0000 mgr.smithi097.swvonl (mgr.14180) 2256 : cluster [DBG] pgmap v1587: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.505 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:54 smithi097 bash[24958]: cluster 2023-11-17T07:38:52.473345+0000 mgr.smithi097.swvonl (mgr.14180) 2257 : cluster [DBG] pgmap v1588: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.505 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:56 smithi097 bash[24958]: cluster 2023-11-17T07:38:54.474434+0000 mgr.smithi097.swvonl (mgr.14180) 2258 : cluster [DBG] pgmap v1589: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.505 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:56.445571+0000 mon.smithi097 (mon.0) 1238 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:40:05.505 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:56.716868+0000 mon.smithi097 (mon.0) 1239 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.513 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:56.990796+0000 mon.smithi097 (mon.0) 1240 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:40:05.513 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:56.992178+0000 mon.smithi097 (mon.0) 1241 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:40:05.513 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:56.998734+0000 mon.smithi097 (mon.0) 1242 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.513 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:57.005312+0000 mon.smithi097 (mon.0) 1243 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:57 smithi097 bash[24958]: audit 2023-11-17T07:38:57.018280+0000 mon.smithi097 (mon.0) 1244 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:58 smithi097 bash[24958]: cluster 2023-11-17T07:38:56.475651+0000 mgr.smithi097.swvonl (mgr.14180) 2259 : cluster [DBG] pgmap v1590: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:58 smithi097 bash[24958]: cluster 2023-11-17T07:38:56.999539+0000 mgr.smithi097.swvonl (mgr.14180) 2260 : cluster [DBG] pgmap v1591: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:58 smithi097 bash[24958]: cephadm 2023-11-17T07:38:57.023167+0000 mgr.smithi097.swvonl (mgr.14180) 2261 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ymtnkn on smithi193 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:58 smithi097 bash[24958]: cluster 2023-11-17T07:38:57.163516+0000 mon.smithi097 (mon.0) 1245 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:58 smithi097 bash[24958]: cluster 2023-11-17T07:38:57.163599+0000 mon.smithi097 (mon.0) 1246 : cluster [INF] Cluster is now healthy 2023-11-17T07:40:05.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:38:59 smithi097 bash[24958]: cluster 2023-11-17T07:38:59.000416+0000 mgr.smithi097.swvonl (mgr.14180) 2262 : cluster [DBG] pgmap v1592: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: cluster 2023-11-17T07:39:01.001569+0000 mgr.smithi097.swvonl (mgr.14180) 2263 : cluster [DBG] pgmap v1593: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: cephadm 2023-11-17T07:39:01.124321+0000 mgr.smithi097.swvonl (mgr.14180) 2264 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.515 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.516 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: cephadm 2023-11-17T07:39:01.124918+0000 mgr.smithi097.swvonl (mgr.14180) 2265 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: audit 2023-11-17T07:39:01.125524+0000 mon.smithi097 (mon.0) 1247 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ymtnkn"}]: dispatch 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: cephadm 2023-11-17T07:39:01.127323+0000 mgr.smithi097.swvonl (mgr.14180) 2266 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ymtnkn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.517 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ymtnkn ... 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:02 smithi097 bash[24958]: cephadm 2023-11-17T07:39:01.130984+0000 mgr.smithi097.swvonl (mgr.14180) 2267 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.pqfllk on smithi097 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:04 smithi097 bash[24958]: cluster 2023-11-17T07:39:03.002325+0000 mgr.smithi097.swvonl (mgr.14180) 2268 : cluster [DBG] pgmap v1594: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:05 smithi097 bash[24958]: audit 2023-11-17T07:39:04.955212+0000 mon.smithi097 (mon.0) 1248 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.pqfllk"}]: dispatch 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cephadm 2023-11-17T07:39:04.953889+0000 mgr.smithi097.swvonl (mgr.14180) 2269 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.518 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:05.519 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.520 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cephadm 2023-11-17T07:39:04.954424+0000 mgr.smithi097.swvonl (mgr.14180) 2270 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cephadm 2023-11-17T07:39:04.956846+0000 mgr.smithi097.swvonl (mgr.14180) 2271 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.pqfllk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.pqfllk ... 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:05.521 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:05.523 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cephadm 2023-11-17T07:39:04.961751+0000 mgr.smithi097.swvonl (mgr.14180) 2272 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.nflyro on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cephadm 2023-11-17T07:39:04.963518+0000 mgr.smithi097.swvonl (mgr.14180) 2273 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cluster 2023-11-17T07:39:04.964743+0000 mgr.smithi097.swvonl (mgr.14180) 2274 : cluster [DBG] pgmap v1595: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: cluster 2023-11-17T07:39:05.077474+0000 mon.smithi097 (mon.0) 1249 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:06 smithi097 bash[24958]: audit 2023-11-17T07:39:05.209800+0000 mon.smithi097 (mon.0) 1250 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:08 smithi097 bash[24958]: cluster 2023-11-17T07:39:06.965717+0000 mgr.smithi097.swvonl (mgr.14180) 2275 : cluster [DBG] pgmap v1596: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 195 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:10 smithi097 bash[24958]: cluster 2023-11-17T07:39:08.966837+0000 mgr.smithi097.swvonl (mgr.14180) 2276 : cluster [DBG] pgmap v1597: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:12 smithi097 bash[24958]: cluster 2023-11-17T07:39:10.968187+0000 mgr.smithi097.swvonl (mgr.14180) 2277 : cluster [DBG] pgmap v1598: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 342 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:14 smithi097 bash[24958]: cluster 2023-11-17T07:39:12.969134+0000 mgr.smithi097.swvonl (mgr.14180) 2278 : cluster [DBG] pgmap v1599: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:16 smithi097 bash[24958]: cluster 2023-11-17T07:39:14.970521+0000 mgr.smithi097.swvonl (mgr.14180) 2279 : cluster [DBG] pgmap v1600: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 171 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:18 smithi097 bash[24958]: cluster 2023-11-17T07:39:16.971878+0000 mgr.smithi097.swvonl (mgr.14180) 2280 : cluster [DBG] pgmap v1601: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.524 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:20 smithi097 bash[24958]: cluster 2023-11-17T07:39:18.972958+0000 mgr.smithi097.swvonl (mgr.14180) 2281 : cluster [DBG] pgmap v1602: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:22 smithi097 bash[24958]: cluster 2023-11-17T07:39:20.974262+0000 mgr.smithi097.swvonl (mgr.14180) 2282 : cluster [DBG] pgmap v1603: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:24 smithi097 bash[24958]: cluster 2023-11-17T07:39:22.975243+0000 mgr.smithi097.swvonl (mgr.14180) 2283 : cluster [DBG] pgmap v1604: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:26 smithi097 bash[24958]: cluster 2023-11-17T07:39:24.975951+0000 mgr.smithi097.swvonl (mgr.14180) 2284 : cluster [DBG] pgmap v1605: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:28 smithi097 bash[24958]: cluster 2023-11-17T07:39:26.976922+0000 mgr.smithi097.swvonl (mgr.14180) 2285 : cluster [DBG] pgmap v1606: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:30 smithi097 bash[24958]: cluster 2023-11-17T07:39:28.978062+0000 mgr.smithi097.swvonl (mgr.14180) 2286 : cluster [DBG] pgmap v1607: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:32 smithi097 bash[24958]: cluster 2023-11-17T07:39:30.979497+0000 mgr.smithi097.swvonl (mgr.14180) 2287 : cluster [DBG] pgmap v1608: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:34 smithi097 bash[24958]: cluster 2023-11-17T07:39:32.980256+0000 mgr.smithi097.swvonl (mgr.14180) 2288 : cluster [DBG] pgmap v1609: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:36 smithi097 bash[24958]: cluster 2023-11-17T07:39:34.981540+0000 mgr.smithi097.swvonl (mgr.14180) 2289 : cluster [DBG] pgmap v1610: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:38 smithi097 bash[24958]: cluster 2023-11-17T07:39:36.982621+0000 mgr.smithi097.swvonl (mgr.14180) 2290 : cluster [DBG] pgmap v1611: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:40 smithi097 bash[24958]: cluster 2023-11-17T07:39:38.983645+0000 mgr.smithi097.swvonl (mgr.14180) 2291 : cluster [DBG] pgmap v1612: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.525 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:42 smithi097 bash[24958]: cluster 2023-11-17T07:39:40.984971+0000 mgr.smithi097.swvonl (mgr.14180) 2292 : cluster [DBG] pgmap v1613: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:44 smithi097 bash[24958]: cluster 2023-11-17T07:39:42.985941+0000 mgr.smithi097.swvonl (mgr.14180) 2293 : cluster [DBG] pgmap v1614: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:46 smithi097 bash[24958]: cluster 2023-11-17T07:39:44.987329+0000 mgr.smithi097.swvonl (mgr.14180) 2294 : cluster [DBG] pgmap v1615: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:48 smithi097 bash[24958]: cluster 2023-11-17T07:39:46.988371+0000 mgr.smithi097.swvonl (mgr.14180) 2295 : cluster [DBG] pgmap v1616: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:50 smithi097 bash[24958]: cluster 2023-11-17T07:39:48.989229+0000 mgr.smithi097.swvonl (mgr.14180) 2296 : cluster [DBG] pgmap v1617: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:52 smithi097 bash[24958]: cluster 2023-11-17T07:39:50.990605+0000 mgr.smithi097.swvonl (mgr.14180) 2297 : cluster [DBG] pgmap v1618: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:53 smithi097 bash[24958]: cluster 2023-11-17T07:39:52.991557+0000 mgr.smithi097.swvonl (mgr.14180) 2298 : cluster [DBG] pgmap v1619: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:56 smithi097 bash[24958]: cluster 2023-11-17T07:39:54.992904+0000 mgr.smithi097.swvonl (mgr.14180) 2299 : cluster [DBG] pgmap v1620: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:39:58 smithi097 bash[24958]: cluster 2023-11-17T07:39:56.993523+0000 mgr.smithi097.swvonl (mgr.14180) 2300 : cluster [DBG] pgmap v1621: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:00 smithi097 bash[24958]: cluster 2023-11-17T07:39:58.994644+0000 mgr.smithi097.swvonl (mgr.14180) 2301 : cluster [DBG] pgmap v1622: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:00 smithi097 bash[24958]: cluster 2023-11-17T07:40:00.000171+0000 mon.smithi097 (mon.0) 1251 : cluster [WRN] overall HEALTH_WARN Failed to place 4 daemon(s) 2023-11-17T07:40:05.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:02 smithi097 bash[24958]: cluster 2023-11-17T07:40:00.996081+0000 mgr.smithi097.swvonl (mgr.14180) 2302 : cluster [DBG] pgmap v1623: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:05.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:04 smithi097 bash[24958]: cluster 2023-11-17T07:40:02.997237+0000 mgr.smithi097.swvonl (mgr.14180) 2303 : cluster [DBG] pgmap v1624: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:05.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:05 smithi097 bash[24958]: audit 2023-11-17T07:40:04.968143+0000 mon.smithi097 (mon.0) 1252 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:40:05.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:05 smithi193 bash[28559]: audit 2023-11-17T07:40:04.968143+0000 mon.smithi097 (mon.0) 1252 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:40:06.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:06.447 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:06 smithi097 bash[24958]: cluster 2023-11-17T07:40:04.998685+0000 mgr.smithi097.swvonl (mgr.14180) 2304 : cluster [DBG] pgmap v1625: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:06.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:06 smithi193 bash[28559]: cluster 2023-11-17T07:40:04.998685+0000 mgr.smithi097.swvonl (mgr.14180) 2304 : cluster [DBG] pgmap v1625: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:08 smithi097 bash[24958]: cluster 2023-11-17T07:40:06.999827+0000 mgr.smithi097.swvonl (mgr.14180) 2305 : cluster [DBG] pgmap v1626: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:08.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:08 smithi193 bash[28559]: cluster 2023-11-17T07:40:06.999827+0000 mgr.smithi097.swvonl (mgr.14180) 2305 : cluster [DBG] pgmap v1626: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:10.407 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:10.407 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:29:40.349837Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:29:39.751340Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:29:40.350048Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:37:56.440441Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.blimzu on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:39:01.127123Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ymtnkn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn\nDeploy daemon haproxy.nfs.foo.smithi193.ymtnkn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:39:04.956559Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pqfllk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk\nDeploy daemon haproxy.nfs.foo.smithi097.pqfllk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:39:04.961555Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nflyro on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:29:39.753296Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:29:39.751693Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:29:39.751936Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:29:39.753516Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:29:39.752163Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:29:39.752387Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:29:40.350295Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:10.462 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:10 smithi097 bash[24958]: cluster 2023-11-17T07:40:09.000956+0000 mgr.smithi097.swvonl (mgr.14180) 2306 : cluster [DBG] pgmap v1627: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:10 smithi193 bash[28559]: cluster 2023-11-17T07:40:09.000956+0000 mgr.smithi097.swvonl (mgr.14180) 2306 : cluster [DBG] pgmap v1627: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:11.398 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:10.394809+0000 mgr.smithi097.swvonl (mgr.14180) 2307 : audit [DBG] from='client.15608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:10.735265+0000 mon.smithi097 (mon.0) 1253 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: cluster 2023-11-17T07:40:11.002172+0000 mgr.smithi097.swvonl (mgr.14180) 2308 : cluster [DBG] pgmap v1628: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.039491+0000 mon.smithi097 (mon.0) 1254 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.308140+0000 mon.smithi097 (mon.0) 1255 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:40:12.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.309539+0000 mon.smithi097 (mon.0) 1256 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:40:12.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.316639+0000 mon.smithi097 (mon.0) 1257 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.323720+0000 mon.smithi097 (mon.0) 1258 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:40:12.050 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:11 smithi097 bash[24958]: audit 2023-11-17T07:40:11.335384+0000 mon.smithi097 (mon.0) 1259 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:10.394809+0000 mgr.smithi097.swvonl (mgr.14180) 2307 : audit [DBG] from='client.15608 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:12.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:10.735265+0000 mon.smithi097 (mon.0) 1253 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: cluster 2023-11-17T07:40:11.002172+0000 mgr.smithi097.swvonl (mgr.14180) 2308 : cluster [DBG] pgmap v1628: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.039491+0000 mon.smithi097 (mon.0) 1254 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.308140+0000 mon.smithi097 (mon.0) 1255 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.309539+0000 mon.smithi097 (mon.0) 1256 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.316639+0000 mon.smithi097 (mon.0) 1257 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.323720+0000 mon.smithi097 (mon.0) 1258 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:40:12.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:11 smithi193 bash[28559]: audit 2023-11-17T07:40:11.335384+0000 mon.smithi097 (mon.0) 1259 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:12.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:13.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:12 smithi097 bash[24958]: cluster 2023-11-17T07:40:11.317317+0000 mgr.smithi097.swvonl (mgr.14180) 2309 : cluster [DBG] pgmap v1629: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:13.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:12 smithi097 bash[24958]: cephadm 2023-11-17T07:40:11.338851+0000 mgr.smithi097.swvonl (mgr.14180) 2310 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.snvmbr on smithi193 2023-11-17T07:40:13.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:12 smithi097 bash[24958]: cluster 2023-11-17T07:40:11.734271+0000 mon.smithi097 (mon.0) 1260 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:40:13.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:12 smithi097 bash[24958]: cluster 2023-11-17T07:40:11.734333+0000 mon.smithi097 (mon.0) 1261 : cluster [INF] Cluster is now healthy 2023-11-17T07:40:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:12 smithi193 bash[28559]: cluster 2023-11-17T07:40:11.317317+0000 mgr.smithi097.swvonl (mgr.14180) 2309 : cluster [DBG] pgmap v1629: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:12 smithi193 bash[28559]: cephadm 2023-11-17T07:40:11.338851+0000 mgr.smithi097.swvonl (mgr.14180) 2310 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.snvmbr on smithi193 2023-11-17T07:40:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:12 smithi193 bash[28559]: cluster 2023-11-17T07:40:11.734271+0000 mon.smithi097 (mon.0) 1260 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:40:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:12 smithi193 bash[28559]: cluster 2023-11-17T07:40:11.734333+0000 mon.smithi097 (mon.0) 1261 : cluster [INF] Cluster is now healthy 2023-11-17T07:40:15.043 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:14 smithi097 bash[24958]: cluster 2023-11-17T07:40:13.318137+0000 mgr.smithi097.swvonl (mgr.14180) 2311 : cluster [DBG] pgmap v1630: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:15.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:14 smithi193 bash[28559]: cluster 2023-11-17T07:40:13.318137+0000 mgr.smithi097.swvonl (mgr.14180) 2311 : cluster [DBG] pgmap v1630: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:16.049 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:15 smithi097 bash[24958]: audit 2023-11-17T07:40:15.444600+0000 mon.smithi097 (mon.0) 1262 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.snvmbr"}]: dispatch 2023-11-17T07:40:16.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:15 smithi193 bash[28559]: audit 2023-11-17T07:40:15.444600+0000 mon.smithi097 (mon.0) 1262 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.snvmbr"}]: dispatch 2023-11-17T07:40:16.673 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:16.674 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:39:01.127123Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ymtnkn on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ymtnkn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ymtnkn\nDeploy daemon haproxy.nfs.foo.smithi193.ymtnkn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:39:04.956559Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.pqfllk on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-pqfllk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.pqfllk\nDeploy daemon haproxy.nfs.foo.smithi097.pqfllk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:39:04.961555Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nflyro on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:15.445916Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\nDeploy daemon haproxy.nfs.foo.smithi193.snvmbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: cluster 2023-11-17T07:40:15.319273+0000 mgr.smithi097.swvonl (mgr.14180) 2312 : cluster [DBG] pgmap v1631: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: cephadm 2023-11-17T07:40:15.443377+0000 mgr.smithi097.swvonl (mgr.14180) 2313 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: cephadm 2023-11-17T07:40:15.443861+0000 mgr.smithi097.swvonl (mgr.14180) 2314 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: cephadm 2023-11-17T07:40:15.446113+0000 mgr.smithi097.swvonl (mgr.14180) 2315 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:16 smithi097 bash[24958]: cephadm 2023-11-17T07:40:15.449458+0000 mgr.smithi097.swvonl (mgr.14180) 2316 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ghcnuy on smithi097 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: cluster 2023-11-17T07:40:15.319273+0000 mgr.smithi097.swvonl (mgr.14180) 2312 : cluster [DBG] pgmap v1631: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: cephadm 2023-11-17T07:40:15.443377+0000 mgr.smithi097.swvonl (mgr.14180) 2313 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: cephadm 2023-11-17T07:40:15.443861+0000 mgr.smithi097.swvonl (mgr.14180) 2314 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: cephadm 2023-11-17T07:40:15.446113+0000 mgr.smithi097.swvonl (mgr.14180) 2315 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.snvmbr ... 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:17.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:16 smithi193 bash[28559]: cephadm 2023-11-17T07:40:15.449458+0000 mgr.smithi097.swvonl (mgr.14180) 2316 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.ghcnuy on smithi097 2023-11-17T07:40:17.657 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:18.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:18 smithi097 bash[24958]: audit 2023-11-17T07:40:16.666576+0000 mgr.smithi097.swvonl (mgr.14180) 2317 : audit [DBG] from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:18.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:18 smithi193 bash[28559]: audit 2023-11-17T07:40:16.666576+0000 mgr.smithi097.swvonl (mgr.14180) 2317 : audit [DBG] from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:18.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:19.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:19 smithi097 bash[24958]: cluster 2023-11-17T07:40:17.320001+0000 mgr.smithi097.swvonl (mgr.14180) 2318 : cluster [DBG] pgmap v1632: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:19 smithi193 bash[28559]: cluster 2023-11-17T07:40:17.320001+0000 mgr.smithi097.swvonl (mgr.14180) 2318 : cluster [DBG] pgmap v1632: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:20.378 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:20 smithi097 bash[24958]: audit 2023-11-17T07:40:19.975300+0000 mon.smithi097 (mon.0) 1263 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ghcnuy"}]: dispatch 2023-11-17T07:40:20.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:20 smithi193 bash[28559]: audit 2023-11-17T07:40:19.975300+0000 mon.smithi097 (mon.0) 1263 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.ghcnuy"}]: dispatch 2023-11-17T07:40:21.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cluster 2023-11-17T07:40:19.320880+0000 mgr.smithi097.swvonl (mgr.14180) 2319 : cluster [DBG] pgmap v1633: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:21.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cephadm 2023-11-17T07:40:19.974468+0000 mgr.smithi097.swvonl (mgr.14180) 2320 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cephadm 2023-11-17T07:40:19.974783+0000 mgr.smithi097.swvonl (mgr.14180) 2321 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cephadm 2023-11-17T07:40:19.976235+0000 mgr.smithi097.swvonl (mgr.14180) 2322 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.555 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cephadm 2023-11-17T07:40:19.978013+0000 mgr.smithi097.swvonl (mgr.14180) 2323 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:21.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cephadm 2023-11-17T07:40:19.979526+0000 mgr.smithi097.swvonl (mgr.14180) 2324 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:21.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cluster 2023-11-17T07:40:19.980775+0000 mgr.smithi097.swvonl (mgr.14180) 2325 : cluster [DBG] pgmap v1634: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2023-11-17T07:40:21.556 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: cluster 2023-11-17T07:40:20.106048+0000 mon.smithi097 (mon.0) 1264 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:40:21.557 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:21 smithi097 bash[24958]: audit 2023-11-17T07:40:20.261594+0000 mon.smithi097 (mon.0) 1265 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:21.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cluster 2023-11-17T07:40:19.320880+0000 mgr.smithi097.swvonl (mgr.14180) 2319 : cluster [DBG] pgmap v1633: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cephadm 2023-11-17T07:40:19.974468+0000 mgr.smithi097.swvonl (mgr.14180) 2320 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:40:21.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cephadm 2023-11-17T07:40:19.974783+0000 mgr.smithi097.swvonl (mgr.14180) 2321 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cephadm 2023-11-17T07:40:19.976235+0000 mgr.smithi097.swvonl (mgr.14180) 2322 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.ghcnuy ... 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cephadm 2023-11-17T07:40:19.978013+0000 mgr.smithi097.swvonl (mgr.14180) 2323 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cephadm 2023-11-17T07:40:19.979526+0000 mgr.smithi097.swvonl (mgr.14180) 2324 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cluster 2023-11-17T07:40:19.980775+0000 mgr.smithi097.swvonl (mgr.14180) 2325 : cluster [DBG] pgmap v1634: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 228 B/s rd, 0 op/s 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: cluster 2023-11-17T07:40:20.106048+0000 mon.smithi097 (mon.0) 1264 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:40:21.562 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:21 smithi193 bash[28559]: audit 2023-11-17T07:40:20.261594+0000 mon.smithi097 (mon.0) 1265 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:40:23.178 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:23.178 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:15.445916Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\nDeploy daemon haproxy.nfs.foo.smithi193.snvmbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.976117Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\nDeploy daemon haproxy.nfs.foo.smithi097.ghcnuy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.977920Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:23.484 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:23 smithi097 bash[24958]: cluster 2023-11-17T07:40:21.981856+0000 mgr.smithi097.swvonl (mgr.14180) 2326 : cluster [DBG] pgmap v1635: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:23.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:23 smithi193 bash[28559]: cluster 2023-11-17T07:40:21.981856+0000 mgr.smithi097.swvonl (mgr.14180) 2326 : cluster [DBG] pgmap v1635: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:24.125 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:24.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:24 smithi097 bash[24958]: audit 2023-11-17T07:40:23.166661+0000 mgr.smithi097.swvonl (mgr.14180) 2327 : audit [DBG] from='client.15616 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:24.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:24 smithi193 bash[28559]: audit 2023-11-17T07:40:23.166661+0000 mgr.smithi097.swvonl (mgr.14180) 2327 : audit [DBG] from='client.15616 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:25.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:25.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:25 smithi097 bash[24958]: cluster 2023-11-17T07:40:23.982960+0000 mgr.smithi097.swvonl (mgr.14180) 2328 : cluster [DBG] pgmap v1636: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:25 smithi193 bash[28559]: cluster 2023-11-17T07:40:23.982960+0000 mgr.smithi097.swvonl (mgr.14180) 2328 : cluster [DBG] pgmap v1636: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:27.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:27 smithi097 bash[24958]: cluster 2023-11-17T07:40:25.984314+0000 mgr.smithi097.swvonl (mgr.14180) 2329 : cluster [DBG] pgmap v1637: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:27 smithi193 bash[28559]: cluster 2023-11-17T07:40:25.984314+0000 mgr.smithi097.swvonl (mgr.14180) 2329 : cluster [DBG] pgmap v1637: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:29.051 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:29.051 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:15.445916Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\nDeploy daemon haproxy.nfs.foo.smithi193.snvmbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.976117Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\nDeploy daemon haproxy.nfs.foo.smithi097.ghcnuy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.977920Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:29.373 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:29 smithi097 bash[24958]: cluster 2023-11-17T07:40:27.984980+0000 mgr.smithi097.swvonl (mgr.14180) 2330 : cluster [DBG] pgmap v1638: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:29.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:29 smithi193 bash[28559]: cluster 2023-11-17T07:40:27.984980+0000 mgr.smithi097.swvonl (mgr.14180) 2330 : cluster [DBG] pgmap v1638: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 192 B/s rd, 0 op/s 2023-11-17T07:40:29.978 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:30.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:30 smithi097 bash[24958]: audit 2023-11-17T07:40:29.044029+0000 mgr.smithi097.swvonl (mgr.14180) 2331 : audit [DBG] from='client.15620 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:30.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:30 smithi193 bash[28559]: audit 2023-11-17T07:40:29.044029+0000 mgr.smithi097.swvonl (mgr.14180) 2331 : audit [DBG] from='client.15620 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:30.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:31.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:31 smithi097 bash[24958]: cluster 2023-11-17T07:40:29.986340+0000 mgr.smithi097.swvonl (mgr.14180) 2332 : cluster [DBG] pgmap v1639: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 384 B/s rd, 0 op/s 2023-11-17T07:40:31.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:31 smithi193 bash[28559]: cluster 2023-11-17T07:40:29.986340+0000 mgr.smithi097.swvonl (mgr.14180) 2332 : cluster [DBG] pgmap v1639: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 384 B/s rd, 0 op/s 2023-11-17T07:40:33.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:33 smithi097 bash[24958]: cluster 2023-11-17T07:40:31.987393+0000 mgr.smithi097.swvonl (mgr.14180) 2333 : cluster [DBG] pgmap v1640: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:33.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:33 smithi193 bash[28559]: cluster 2023-11-17T07:40:31.987393+0000 mgr.smithi097.swvonl (mgr.14180) 2333 : cluster [DBG] pgmap v1640: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:34.953 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:40:34.953 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:15.445916Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\nDeploy daemon haproxy.nfs.foo.smithi193.snvmbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.976117Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\nDeploy daemon haproxy.nfs.foo.smithi097.ghcnuy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.977920Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:40:35.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:35 smithi097 bash[24958]: cluster 2023-11-17T07:40:33.988542+0000 mgr.smithi097.swvonl (mgr.14180) 2334 : cluster [DBG] pgmap v1641: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:35.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:35 smithi193 bash[28559]: cluster 2023-11-17T07:40:33.988542+0000 mgr.smithi097.swvonl (mgr.14180) 2334 : cluster [DBG] pgmap v1641: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:40:35.902 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:40:36.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:36 smithi097 bash[24958]: audit 2023-11-17T07:40:34.946174+0000 mgr.smithi097.swvonl (mgr.14180) 2335 : audit [DBG] from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:36.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:36 smithi193 bash[28559]: audit 2023-11-17T07:40:34.946174+0000 mgr.smithi097.swvonl (mgr.14180) 2335 : audit [DBG] from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:40:36.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:40:37.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:37 smithi097 bash[24958]: cluster 2023-11-17T07:40:35.989840+0000 mgr.smithi097.swvonl (mgr.14180) 2336 : cluster [DBG] pgmap v1642: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.291 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:39 smithi097 bash[24958]: cluster 2023-11-17T07:40:37.990446+0000 mgr.smithi097.swvonl (mgr.14180) 2337 : cluster [DBG] pgmap v1643: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.291 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:41:40.291 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:39:04.963416Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.qusahn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:15.445916Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.snvmbr on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-snvmbr\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.snvmbr\nDeploy daemon haproxy.nfs.foo.smithi193.snvmbr ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.976117Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.ghcnuy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-ghcnuy\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.ghcnuy\nDeploy daemon haproxy.nfs.foo.smithi097.ghcnuy ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:40:19.977920Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.zpfoik on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:41:40.292 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:41 smithi097 bash[24958]: cluster 2023-11-17T07:40:39.991823+0000 mgr.smithi097.swvonl (mgr.14180) 2338 : cluster [DBG] pgmap v1644: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.294 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:37 smithi193 bash[28559]: cluster 2023-11-17T07:40:35.989840+0000 mgr.smithi097.swvonl (mgr.14180) 2336 : cluster [DBG] pgmap v1642: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:39 smithi193 bash[28559]: cluster 2023-11-17T07:40:37.990446+0000 mgr.smithi097.swvonl (mgr.14180) 2337 : cluster [DBG] pgmap v1643: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:41 smithi193 bash[28559]: cluster 2023-11-17T07:40:39.991823+0000 mgr.smithi097.swvonl (mgr.14180) 2338 : cluster [DBG] pgmap v1644: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:42 smithi193 bash[28559]: audit 2023-11-17T07:40:41.044844+0000 mgr.smithi097.swvonl (mgr.14180) 2339 : audit [DBG] from='client.15628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:43 smithi193 bash[28559]: cluster 2023-11-17T07:40:41.992894+0000 mgr.smithi097.swvonl (mgr.14180) 2340 : cluster [DBG] pgmap v1645: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:45 smithi193 bash[28559]: cluster 2023-11-17T07:40:43.993935+0000 mgr.smithi097.swvonl (mgr.14180) 2341 : cluster [DBG] pgmap v1646: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:47 smithi193 bash[28559]: cluster 2023-11-17T07:40:45.995251+0000 mgr.smithi097.swvonl (mgr.14180) 2342 : cluster [DBG] pgmap v1647: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:49 smithi193 bash[28559]: cluster 2023-11-17T07:40:47.995927+0000 mgr.smithi097.swvonl (mgr.14180) 2343 : cluster [DBG] pgmap v1648: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.299 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:51 smithi193 bash[28559]: cluster 2023-11-17T07:40:49.997347+0000 mgr.smithi097.swvonl (mgr.14180) 2344 : cluster [DBG] pgmap v1649: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:53 smithi193 bash[28559]: cluster 2023-11-17T07:40:51.998446+0000 mgr.smithi097.swvonl (mgr.14180) 2345 : cluster [DBG] pgmap v1650: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:55 smithi193 bash[28559]: cluster 2023-11-17T07:40:53.999506+0000 mgr.smithi097.swvonl (mgr.14180) 2346 : cluster [DBG] pgmap v1651: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:56 smithi193 bash[28559]: cluster 2023-11-17T07:40:56.000841+0000 mgr.smithi097.swvonl (mgr.14180) 2347 : cluster [DBG] pgmap v1652: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:40:59 smithi193 bash[28559]: cluster 2023-11-17T07:40:58.001500+0000 mgr.smithi097.swvonl (mgr.14180) 2348 : cluster [DBG] pgmap v1653: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:01 smithi193 bash[28559]: cluster 2023-11-17T07:41:00.002805+0000 mgr.smithi097.swvonl (mgr.14180) 2349 : cluster [DBG] pgmap v1654: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:03 smithi193 bash[28559]: cluster 2023-11-17T07:41:02.003893+0000 mgr.smithi097.swvonl (mgr.14180) 2350 : cluster [DBG] pgmap v1655: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:05 smithi193 bash[28559]: cluster 2023-11-17T07:41:04.005024+0000 mgr.smithi097.swvonl (mgr.14180) 2351 : cluster [DBG] pgmap v1656: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:07 smithi193 bash[28559]: cluster 2023-11-17T07:41:06.006337+0000 mgr.smithi097.swvonl (mgr.14180) 2352 : cluster [DBG] pgmap v1657: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:09 smithi193 bash[28559]: cluster 2023-11-17T07:41:08.007120+0000 mgr.smithi097.swvonl (mgr.14180) 2353 : cluster [DBG] pgmap v1658: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:11 smithi193 bash[28559]: cluster 2023-11-17T07:41:10.008482+0000 mgr.smithi097.swvonl (mgr.14180) 2354 : cluster [DBG] pgmap v1659: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:13 smithi193 bash[28559]: cluster 2023-11-17T07:41:12.009486+0000 mgr.smithi097.swvonl (mgr.14180) 2355 : cluster [DBG] pgmap v1660: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:15 smithi193 bash[28559]: cluster 2023-11-17T07:41:14.010576+0000 mgr.smithi097.swvonl (mgr.14180) 2356 : cluster [DBG] pgmap v1661: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:17 smithi193 bash[28559]: cluster 2023-11-17T07:41:16.011878+0000 mgr.smithi097.swvonl (mgr.14180) 2357 : cluster [DBG] pgmap v1662: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.300 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:19 smithi193 bash[28559]: cluster 2023-11-17T07:41:18.012736+0000 mgr.smithi097.swvonl (mgr.14180) 2358 : cluster [DBG] pgmap v1663: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:20 smithi193 bash[28559]: audit 2023-11-17T07:41:19.983345+0000 mon.smithi097 (mon.0) 1266 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: cluster 2023-11-17T07:41:20.014276+0000 mgr.smithi097.swvonl (mgr.14180) 2359 : cluster [DBG] pgmap v1664: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.263125+0000 mon.smithi097 (mon.0) 1267 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.538812+0000 mon.smithi097 (mon.0) 1268 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.540842+0000 mon.smithi097 (mon.0) 1269 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.551507+0000 mon.smithi097 (mon.0) 1270 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.560504+0000 mon.smithi097 (mon.0) 1271 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:21 smithi193 bash[28559]: audit 2023-11-17T07:41:20.576423+0000 mon.smithi097 (mon.0) 1272 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:22 smithi193 bash[28559]: cluster 2023-11-17T07:41:20.552482+0000 mgr.smithi097.swvonl (mgr.14180) 2360 : cluster [DBG] pgmap v1665: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:22 smithi193 bash[28559]: cephadm 2023-11-17T07:41:20.580292+0000 mgr.smithi097.swvonl (mgr.14180) 2361 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.hczihz on smithi193 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:22 smithi193 bash[28559]: cluster 2023-11-17T07:41:21.547822+0000 mon.smithi097 (mon.0) 1273 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:22 smithi193 bash[28559]: cluster 2023-11-17T07:41:21.547884+0000 mon.smithi097 (mon.0) 1274 : cluster [INF] Cluster is now healthy 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:24 smithi193 bash[28559]: cluster 2023-11-17T07:41:22.553198+0000 mgr.smithi097.swvonl (mgr.14180) 2362 : cluster [DBG] pgmap v1666: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.301 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:25 smithi193 bash[28559]: audit 2023-11-17T07:41:24.494105+0000 mon.smithi097 (mon.0) 1275 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.hczihz"}]: dispatch 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: cephadm 2023-11-17T07:41:24.493032+0000 mgr.smithi097.swvonl (mgr.14180) 2363 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.302 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:41:40.303 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.304 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: cephadm 2023-11-17T07:41:24.493475+0000 mgr.smithi097.swvonl (mgr.14180) 2364 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: cephadm 2023-11-17T07:41:24.495802+0000 mgr.smithi097.swvonl (mgr.14180) 2365 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: cephadm 2023-11-17T07:41:24.501025+0000 mgr.smithi097.swvonl (mgr.14180) 2366 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.mqptos on smithi097 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:26 smithi193 bash[28559]: cluster 2023-11-17T07:41:24.554031+0000 mgr.smithi097.swvonl (mgr.14180) 2367 : cluster [DBG] pgmap v1667: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:27 smithi193 bash[28559]: cluster 2023-11-17T07:41:26.555015+0000 mgr.smithi097.swvonl (mgr.14180) 2368 : cluster [DBG] pgmap v1668: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:27 smithi193 bash[28559]: audit 2023-11-17T07:41:26.582023+0000 mon.smithi097 (mon.0) 1276 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:28 smithi193 bash[28559]: audit 2023-11-17T07:41:28.263733+0000 mon.smithi097 (mon.0) 1277 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.mqptos"}]: dispatch 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cephadm 2023-11-17T07:41:28.262510+0000 mgr.smithi097.swvonl (mgr.14180) 2369 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:41:40.307 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.405 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cephadm 2023-11-17T07:41:28.263090+0000 mgr.smithi097.swvonl (mgr.14180) 2370 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cephadm 2023-11-17T07:41:28.265345+0000 mgr.smithi097.swvonl (mgr.14180) 2371 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.406 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cephadm 2023-11-17T07:41:28.267626+0000 mgr.smithi097.swvonl (mgr.14180) 2372 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cephadm 2023-11-17T07:41:28.269292+0000 mgr.smithi097.swvonl (mgr.14180) 2373 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cluster 2023-11-17T07:41:28.270293+0000 mgr.smithi097.swvonl (mgr.14180) 2374 : cluster [DBG] pgmap v1669: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:29 smithi193 bash[28559]: cluster 2023-11-17T07:41:28.582379+0000 mon.smithi097 (mon.0) 1278 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:31 smithi193 bash[28559]: cluster 2023-11-17T07:41:30.271612+0000 mgr.smithi097.swvonl (mgr.14180) 2375 : cluster [DBG] pgmap v1670: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:31 smithi193 bash[28559]: audit 2023-11-17T07:41:30.314187+0000 mon.smithi097 (mon.0) 1279 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:34 smithi193 bash[28559]: cluster 2023-11-17T07:41:32.272478+0000 mgr.smithi097.swvonl (mgr.14180) 2376 : cluster [DBG] pgmap v1671: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:36 smithi193 bash[28559]: cluster 2023-11-17T07:41:34.273542+0000 mgr.smithi097.swvonl (mgr.14180) 2377 : cluster [DBG] pgmap v1672: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.407 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:38 smithi193 bash[28559]: cluster 2023-11-17T07:41:36.274835+0000 mgr.smithi097.swvonl (mgr.14180) 2378 : cluster [DBG] pgmap v1673: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:42 smithi097 bash[24958]: audit 2023-11-17T07:40:41.044844+0000 mgr.smithi097.swvonl (mgr.14180) 2339 : audit [DBG] from='client.15628 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:43 smithi097 bash[24958]: cluster 2023-11-17T07:40:41.992894+0000 mgr.smithi097.swvonl (mgr.14180) 2340 : cluster [DBG] pgmap v1645: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:45 smithi097 bash[24958]: cluster 2023-11-17T07:40:43.993935+0000 mgr.smithi097.swvonl (mgr.14180) 2341 : cluster [DBG] pgmap v1646: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:47 smithi097 bash[24958]: cluster 2023-11-17T07:40:45.995251+0000 mgr.smithi097.swvonl (mgr.14180) 2342 : cluster [DBG] pgmap v1647: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:49 smithi097 bash[24958]: cluster 2023-11-17T07:40:47.995927+0000 mgr.smithi097.swvonl (mgr.14180) 2343 : cluster [DBG] pgmap v1648: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.408 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:51 smithi097 bash[24958]: cluster 2023-11-17T07:40:49.997347+0000 mgr.smithi097.swvonl (mgr.14180) 2344 : cluster [DBG] pgmap v1649: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:53 smithi097 bash[24958]: cluster 2023-11-17T07:40:51.998446+0000 mgr.smithi097.swvonl (mgr.14180) 2345 : cluster [DBG] pgmap v1650: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:55 smithi097 bash[24958]: cluster 2023-11-17T07:40:53.999506+0000 mgr.smithi097.swvonl (mgr.14180) 2346 : cluster [DBG] pgmap v1651: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:56 smithi097 bash[24958]: cluster 2023-11-17T07:40:56.000841+0000 mgr.smithi097.swvonl (mgr.14180) 2347 : cluster [DBG] pgmap v1652: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:40:59 smithi097 bash[24958]: cluster 2023-11-17T07:40:58.001500+0000 mgr.smithi097.swvonl (mgr.14180) 2348 : cluster [DBG] pgmap v1653: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:01 smithi097 bash[24958]: cluster 2023-11-17T07:41:00.002805+0000 mgr.smithi097.swvonl (mgr.14180) 2349 : cluster [DBG] pgmap v1654: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:03 smithi097 bash[24958]: cluster 2023-11-17T07:41:02.003893+0000 mgr.smithi097.swvonl (mgr.14180) 2350 : cluster [DBG] pgmap v1655: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:05 smithi097 bash[24958]: cluster 2023-11-17T07:41:04.005024+0000 mgr.smithi097.swvonl (mgr.14180) 2351 : cluster [DBG] pgmap v1656: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:07 smithi097 bash[24958]: cluster 2023-11-17T07:41:06.006337+0000 mgr.smithi097.swvonl (mgr.14180) 2352 : cluster [DBG] pgmap v1657: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:09 smithi097 bash[24958]: cluster 2023-11-17T07:41:08.007120+0000 mgr.smithi097.swvonl (mgr.14180) 2353 : cluster [DBG] pgmap v1658: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:11 smithi097 bash[24958]: cluster 2023-11-17T07:41:10.008482+0000 mgr.smithi097.swvonl (mgr.14180) 2354 : cluster [DBG] pgmap v1659: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:13 smithi097 bash[24958]: cluster 2023-11-17T07:41:12.009486+0000 mgr.smithi097.swvonl (mgr.14180) 2355 : cluster [DBG] pgmap v1660: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:15 smithi097 bash[24958]: cluster 2023-11-17T07:41:14.010576+0000 mgr.smithi097.swvonl (mgr.14180) 2356 : cluster [DBG] pgmap v1661: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:17 smithi097 bash[24958]: cluster 2023-11-17T07:41:16.011878+0000 mgr.smithi097.swvonl (mgr.14180) 2357 : cluster [DBG] pgmap v1662: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.409 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:19 smithi097 bash[24958]: cluster 2023-11-17T07:41:18.012736+0000 mgr.smithi097.swvonl (mgr.14180) 2358 : cluster [DBG] pgmap v1663: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:20 smithi097 bash[24958]: audit 2023-11-17T07:41:19.983345+0000 mon.smithi097 (mon.0) 1266 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: cluster 2023-11-17T07:41:20.014276+0000 mgr.smithi097.swvonl (mgr.14180) 2359 : cluster [DBG] pgmap v1664: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.263125+0000 mon.smithi097 (mon.0) 1267 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.538812+0000 mon.smithi097 (mon.0) 1268 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.540842+0000 mon.smithi097 (mon.0) 1269 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.551507+0000 mon.smithi097 (mon.0) 1270 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.560504+0000 mon.smithi097 (mon.0) 1271 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:21 smithi097 bash[24958]: audit 2023-11-17T07:41:20.576423+0000 mon.smithi097 (mon.0) 1272 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:22 smithi097 bash[24958]: cluster 2023-11-17T07:41:20.552482+0000 mgr.smithi097.swvonl (mgr.14180) 2360 : cluster [DBG] pgmap v1665: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:22 smithi097 bash[24958]: cephadm 2023-11-17T07:41:20.580292+0000 mgr.smithi097.swvonl (mgr.14180) 2361 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.hczihz on smithi193 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:22 smithi097 bash[24958]: cluster 2023-11-17T07:41:21.547822+0000 mon.smithi097 (mon.0) 1273 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:22 smithi097 bash[24958]: cluster 2023-11-17T07:41:21.547884+0000 mon.smithi097 (mon.0) 1274 : cluster [INF] Cluster is now healthy 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:24 smithi097 bash[24958]: cluster 2023-11-17T07:41:22.553198+0000 mgr.smithi097.swvonl (mgr.14180) 2362 : cluster [DBG] pgmap v1666: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.410 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:25 smithi097 bash[24958]: audit 2023-11-17T07:41:24.494105+0000 mon.smithi097 (mon.0) 1275 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.hczihz"}]: dispatch 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: cephadm 2023-11-17T07:41:24.493032+0000 mgr.smithi097.swvonl (mgr.14180) 2363 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.411 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.412 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: cephadm 2023-11-17T07:41:24.493475+0000 mgr.smithi097.swvonl (mgr.14180) 2364 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: cephadm 2023-11-17T07:41:24.495802+0000 mgr.smithi097.swvonl (mgr.14180) 2365 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.hczihz ... 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.413 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: cephadm 2023-11-17T07:41:24.501025+0000 mgr.smithi097.swvonl (mgr.14180) 2366 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.mqptos on smithi097 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:26 smithi097 bash[24958]: cluster 2023-11-17T07:41:24.554031+0000 mgr.smithi097.swvonl (mgr.14180) 2367 : cluster [DBG] pgmap v1667: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:27 smithi097 bash[24958]: cluster 2023-11-17T07:41:26.555015+0000 mgr.smithi097.swvonl (mgr.14180) 2368 : cluster [DBG] pgmap v1668: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 194 B/s rd, 0 op/s 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:27 smithi097 bash[24958]: audit 2023-11-17T07:41:26.582023+0000 mon.smithi097 (mon.0) 1276 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:28 smithi097 bash[24958]: audit 2023-11-17T07:41:28.263733+0000 mon.smithi097 (mon.0) 1277 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.mqptos"}]: dispatch 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cephadm 2023-11-17T07:41:28.262510+0000 mgr.smithi097.swvonl (mgr.14180) 2369 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.414 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.415 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cephadm 2023-11-17T07:41:28.263090+0000 mgr.smithi097.swvonl (mgr.14180) 2370 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cephadm 2023-11-17T07:41:28.265345+0000 mgr.smithi097.swvonl (mgr.14180) 2371 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.416 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.mqptos ... 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cephadm 2023-11-17T07:41:28.267626+0000 mgr.smithi097.swvonl (mgr.14180) 2372 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cephadm 2023-11-17T07:41:28.269292+0000 mgr.smithi097.swvonl (mgr.14180) 2373 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cluster 2023-11-17T07:41:28.270293+0000 mgr.smithi097.swvonl (mgr.14180) 2374 : cluster [DBG] pgmap v1669: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:29 smithi097 bash[24958]: cluster 2023-11-17T07:41:28.582379+0000 mon.smithi097 (mon.0) 1278 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:41:40.417 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:31 smithi097 bash[24958]: cluster 2023-11-17T07:41:30.271612+0000 mgr.smithi097.swvonl (mgr.14180) 2375 : cluster [DBG] pgmap v1670: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:41:40.418 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:31 smithi097 bash[24958]: audit 2023-11-17T07:41:30.314187+0000 mon.smithi097 (mon.0) 1279 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:40.418 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:34 smithi097 bash[24958]: cluster 2023-11-17T07:41:32.272478+0000 mgr.smithi097.swvonl (mgr.14180) 2376 : cluster [DBG] pgmap v1671: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.459 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:36 smithi097 bash[24958]: cluster 2023-11-17T07:41:34.273542+0000 mgr.smithi097.swvonl (mgr.14180) 2377 : cluster [DBG] pgmap v1672: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:38 smithi097 bash[24958]: cluster 2023-11-17T07:41:36.274835+0000 mgr.smithi097.swvonl (mgr.14180) 2378 : cluster [DBG] pgmap v1673: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:40 smithi097 bash[24958]: cluster 2023-11-17T07:41:38.275768+0000 mgr.smithi097.swvonl (mgr.14180) 2379 : cluster [DBG] pgmap v1674: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:40.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:40 smithi193 bash[28559]: cluster 2023-11-17T07:41:38.275768+0000 mgr.smithi097.swvonl (mgr.14180) 2379 : cluster [DBG] pgmap v1674: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:41:41.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:41:42.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:42 smithi097 bash[24958]: cluster 2023-11-17T07:41:40.277124+0000 mgr.smithi097.swvonl (mgr.14180) 2380 : cluster [DBG] pgmap v1675: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:42.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:42 smithi097 bash[24958]: audit 2023-11-17T07:41:41.583976+0000 mon.smithi097 (mon.0) 1280 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:42.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:42 smithi193 bash[28559]: cluster 2023-11-17T07:41:40.277124+0000 mgr.smithi097.swvonl (mgr.14180) 2380 : cluster [DBG] pgmap v1675: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:41:42.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:42 smithi193 bash[28559]: audit 2023-11-17T07:41:41.583976+0000 mon.smithi097 (mon.0) 1280 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:41:44.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:44 smithi097 bash[24958]: cluster 2023-11-17T07:41:42.277993+0000 mgr.smithi097.swvonl (mgr.14180) 2381 : cluster [DBG] pgmap v1676: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:44.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:44 smithi193 bash[28559]: cluster 2023-11-17T07:41:42.277993+0000 mgr.smithi097.swvonl (mgr.14180) 2381 : cluster [DBG] pgmap v1676: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:45.405 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:41:45.406 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:24.495507Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\nDeploy daemon haproxy.nfs.foo.smithi193.hczihz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.265073Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\nDeploy daemon haproxy.nfs.foo.smithi097.mqptos ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.267526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:41:46.448 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:46 smithi097 bash[24958]: cluster 2023-11-17T07:41:44.279035+0000 mgr.smithi097.swvonl (mgr.14180) 2382 : cluster [DBG] pgmap v1677: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:46.450 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:41:46.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:46 smithi193 bash[28559]: cluster 2023-11-17T07:41:44.279035+0000 mgr.smithi097.swvonl (mgr.14180) 2382 : cluster [DBG] pgmap v1677: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:47.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:41:47.463 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:47 smithi097 bash[24958]: audit 2023-11-17T07:41:45.393614+0000 mgr.smithi097.swvonl (mgr.14180) 2383 : audit [DBG] from='client.15632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:41:47.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:47 smithi193 bash[28559]: audit 2023-11-17T07:41:45.393614+0000 mgr.smithi097.swvonl (mgr.14180) 2383 : audit [DBG] from='client.15632 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:41:48.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:48 smithi097 bash[24958]: cluster 2023-11-17T07:41:46.279904+0000 mgr.smithi097.swvonl (mgr.14180) 2384 : cluster [DBG] pgmap v1678: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:48.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:48 smithi193 bash[28559]: cluster 2023-11-17T07:41:46.279904+0000 mgr.smithi097.swvonl (mgr.14180) 2384 : cluster [DBG] pgmap v1678: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:41:50.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:50 smithi097 bash[24958]: cluster 2023-11-17T07:41:48.280847+0000 mgr.smithi097.swvonl (mgr.14180) 2385 : cluster [DBG] pgmap v1679: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.547 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:20.547 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:24.495507Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\nDeploy daemon haproxy.nfs.foo.smithi193.hczihz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.265073Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\nDeploy daemon haproxy.nfs.foo.smithi097.mqptos ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.267526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:20.548 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:20.549 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:50 smithi193 bash[28559]: cluster 2023-11-17T07:41:48.280847+0000 mgr.smithi097.swvonl (mgr.14180) 2385 : cluster [DBG] pgmap v1679: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.549 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:52 smithi193 bash[28559]: cluster 2023-11-17T07:41:50.282322+0000 mgr.smithi097.swvonl (mgr.14180) 2386 : cluster [DBG] pgmap v1680: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.549 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:53 smithi193 bash[28559]: audit 2023-11-17T07:41:51.274284+0000 mgr.smithi097.swvonl (mgr.14180) 2387 : audit [DBG] from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:54 smithi193 bash[28559]: cluster 2023-11-17T07:41:52.283369+0000 mgr.smithi097.swvonl (mgr.14180) 2388 : cluster [DBG] pgmap v1681: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:56 smithi193 bash[28559]: cluster 2023-11-17T07:41:54.284603+0000 mgr.smithi097.swvonl (mgr.14180) 2389 : cluster [DBG] pgmap v1682: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:41:58 smithi193 bash[28559]: cluster 2023-11-17T07:41:56.286099+0000 mgr.smithi097.swvonl (mgr.14180) 2390 : cluster [DBG] pgmap v1683: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:00 smithi193 bash[28559]: cluster 2023-11-17T07:41:58.287195+0000 mgr.smithi097.swvonl (mgr.14180) 2391 : cluster [DBG] pgmap v1684: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:02 smithi193 bash[28559]: cluster 2023-11-17T07:42:00.288746+0000 mgr.smithi097.swvonl (mgr.14180) 2392 : cluster [DBG] pgmap v1685: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:04 smithi193 bash[28559]: cluster 2023-11-17T07:42:02.289780+0000 mgr.smithi097.swvonl (mgr.14180) 2393 : cluster [DBG] pgmap v1686: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:06 smithi193 bash[28559]: cluster 2023-11-17T07:42:04.291048+0000 mgr.smithi097.swvonl (mgr.14180) 2394 : cluster [DBG] pgmap v1687: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:08 smithi193 bash[28559]: cluster 2023-11-17T07:42:06.292508+0000 mgr.smithi097.swvonl (mgr.14180) 2395 : cluster [DBG] pgmap v1688: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:10 smithi193 bash[28559]: cluster 2023-11-17T07:42:08.293592+0000 mgr.smithi097.swvonl (mgr.14180) 2396 : cluster [DBG] pgmap v1689: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:12 smithi193 bash[28559]: cluster 2023-11-17T07:42:10.295100+0000 mgr.smithi097.swvonl (mgr.14180) 2397 : cluster [DBG] pgmap v1690: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:13 smithi193 bash[28559]: cluster 2023-11-17T07:42:12.296202+0000 mgr.smithi097.swvonl (mgr.14180) 2398 : cluster [DBG] pgmap v1691: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:15 smithi193 bash[28559]: cluster 2023-11-17T07:42:14.297384+0000 mgr.smithi097.swvonl (mgr.14180) 2399 : cluster [DBG] pgmap v1692: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.550 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:18 smithi193 bash[28559]: cluster 2023-11-17T07:42:16.298844+0000 mgr.smithi097.swvonl (mgr.14180) 2400 : cluster [DBG] pgmap v1693: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:52 smithi097 bash[24958]: cluster 2023-11-17T07:41:50.282322+0000 mgr.smithi097.swvonl (mgr.14180) 2386 : cluster [DBG] pgmap v1680: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:53 smithi097 bash[24958]: audit 2023-11-17T07:41:51.274284+0000 mgr.smithi097.swvonl (mgr.14180) 2387 : audit [DBG] from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:20.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:54 smithi097 bash[24958]: cluster 2023-11-17T07:41:52.283369+0000 mgr.smithi097.swvonl (mgr.14180) 2388 : cluster [DBG] pgmap v1681: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:56 smithi097 bash[24958]: cluster 2023-11-17T07:41:54.284603+0000 mgr.smithi097.swvonl (mgr.14180) 2389 : cluster [DBG] pgmap v1682: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:41:58 smithi097 bash[24958]: cluster 2023-11-17T07:41:56.286099+0000 mgr.smithi097.swvonl (mgr.14180) 2390 : cluster [DBG] pgmap v1683: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:00 smithi097 bash[24958]: cluster 2023-11-17T07:41:58.287195+0000 mgr.smithi097.swvonl (mgr.14180) 2391 : cluster [DBG] pgmap v1684: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:02 smithi097 bash[24958]: cluster 2023-11-17T07:42:00.288746+0000 mgr.smithi097.swvonl (mgr.14180) 2392 : cluster [DBG] pgmap v1685: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:04 smithi097 bash[24958]: cluster 2023-11-17T07:42:02.289780+0000 mgr.smithi097.swvonl (mgr.14180) 2393 : cluster [DBG] pgmap v1686: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:06 smithi097 bash[24958]: cluster 2023-11-17T07:42:04.291048+0000 mgr.smithi097.swvonl (mgr.14180) 2394 : cluster [DBG] pgmap v1687: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:08 smithi097 bash[24958]: cluster 2023-11-17T07:42:06.292508+0000 mgr.smithi097.swvonl (mgr.14180) 2395 : cluster [DBG] pgmap v1688: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:10 smithi097 bash[24958]: cluster 2023-11-17T07:42:08.293592+0000 mgr.smithi097.swvonl (mgr.14180) 2396 : cluster [DBG] pgmap v1689: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:12 smithi097 bash[24958]: cluster 2023-11-17T07:42:10.295100+0000 mgr.smithi097.swvonl (mgr.14180) 2397 : cluster [DBG] pgmap v1690: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:13 smithi097 bash[24958]: cluster 2023-11-17T07:42:12.296202+0000 mgr.smithi097.swvonl (mgr.14180) 2398 : cluster [DBG] pgmap v1691: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:15 smithi097 bash[24958]: cluster 2023-11-17T07:42:14.297384+0000 mgr.smithi097.swvonl (mgr.14180) 2399 : cluster [DBG] pgmap v1692: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:18 smithi097 bash[24958]: cluster 2023-11-17T07:42:16.298844+0000 mgr.smithi097.swvonl (mgr.14180) 2400 : cluster [DBG] pgmap v1693: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:20 smithi097 bash[24958]: cluster 2023-11-17T07:42:18.299857+0000 mgr.smithi097.swvonl (mgr.14180) 2401 : cluster [DBG] pgmap v1694: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:20.574 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:20 smithi193 bash[28559]: cluster 2023-11-17T07:42:18.299857+0000 mgr.smithi097.swvonl (mgr.14180) 2401 : cluster [DBG] pgmap v1694: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:21.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:22 smithi097 bash[24958]: cluster 2023-11-17T07:42:20.301340+0000 mgr.smithi097.swvonl (mgr.14180) 2402 : cluster [DBG] pgmap v1695: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:22.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:22 smithi193 bash[28559]: cluster 2023-11-17T07:42:20.301340+0000 mgr.smithi097.swvonl (mgr.14180) 2402 : cluster [DBG] pgmap v1695: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:24.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:24 smithi097 bash[24958]: cluster 2023-11-17T07:42:22.302014+0000 mgr.smithi097.swvonl (mgr.14180) 2403 : cluster [DBG] pgmap v1696: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:24.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:24 smithi193 bash[28559]: cluster 2023-11-17T07:42:22.302014+0000 mgr.smithi097.swvonl (mgr.14180) 2403 : cluster [DBG] pgmap v1696: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:25.624 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:25.625 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:24.495507Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\nDeploy daemon haproxy.nfs.foo.smithi193.hczihz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.265073Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\nDeploy daemon haproxy.nfs.foo.smithi097.mqptos ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.267526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:26.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:26 smithi097 bash[24958]: cluster 2023-11-17T07:42:24.303126+0000 mgr.smithi097.swvonl (mgr.14180) 2404 : cluster [DBG] pgmap v1697: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:26.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:26 smithi193 bash[28559]: cluster 2023-11-17T07:42:24.303126+0000 mgr.smithi097.swvonl (mgr.14180) 2404 : cluster [DBG] pgmap v1697: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:26.585 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:27.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:27 smithi097 bash[24958]: audit 2023-11-17T07:42:25.613270+0000 mgr.smithi097.swvonl (mgr.14180) 2405 : audit [DBG] from='client.15640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:27 smithi193 bash[28559]: audit 2023-11-17T07:42:25.613270+0000 mgr.smithi097.swvonl (mgr.14180) 2405 : audit [DBG] from='client.15640 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:27.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:28.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:28 smithi097 bash[24958]: cluster 2023-11-17T07:42:26.303953+0000 mgr.smithi097.swvonl (mgr.14180) 2406 : cluster [DBG] pgmap v1698: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:28.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:28 smithi193 bash[28559]: cluster 2023-11-17T07:42:26.303953+0000 mgr.smithi097.swvonl (mgr.14180) 2406 : cluster [DBG] pgmap v1698: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.273677+0000 mon.smithi097 (mon.0) 1281 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.579380+0000 mon.smithi097 (mon.0) 1282 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.580790+0000 mon.smithi097 (mon.0) 1283 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.586886+0000 mon.smithi097 (mon.0) 1284 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.592611+0000 mon.smithi097 (mon.0) 1285 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:42:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:29 smithi097 bash[24958]: audit 2023-11-17T07:42:28.604962+0000 mon.smithi097 (mon.0) 1286 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:29.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.273677+0000 mon.smithi097 (mon.0) 1281 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:42:29.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.579380+0000 mon.smithi097 (mon.0) 1282 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:42:29.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.580790+0000 mon.smithi097 (mon.0) 1283 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:42:29.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.586886+0000 mon.smithi097 (mon.0) 1284 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:29.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.592611+0000 mon.smithi097 (mon.0) 1285 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:42:29.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:29 smithi193 bash[28559]: audit 2023-11-17T07:42:28.604962+0000 mon.smithi097 (mon.0) 1286 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:30 smithi097 bash[24958]: cluster 2023-11-17T07:42:28.304943+0000 mgr.smithi097.swvonl (mgr.14180) 2407 : cluster [DBG] pgmap v1699: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:30 smithi097 bash[24958]: cluster 2023-11-17T07:42:28.587576+0000 mgr.smithi097.swvonl (mgr.14180) 2408 : cluster [DBG] pgmap v1700: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:30 smithi097 bash[24958]: cephadm 2023-11-17T07:42:28.607523+0000 mgr.smithi097.swvonl (mgr.14180) 2409 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.gkfysp on smithi193 2023-11-17T07:42:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:30 smithi097 bash[24958]: cluster 2023-11-17T07:42:29.184019+0000 mon.smithi097 (mon.0) 1287 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:42:30.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:30 smithi097 bash[24958]: cluster 2023-11-17T07:42:29.184094+0000 mon.smithi097 (mon.0) 1288 : cluster [INF] Cluster is now healthy 2023-11-17T07:42:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:30 smithi193 bash[28559]: cluster 2023-11-17T07:42:28.304943+0000 mgr.smithi097.swvonl (mgr.14180) 2407 : cluster [DBG] pgmap v1699: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:30.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:30 smithi193 bash[28559]: cluster 2023-11-17T07:42:28.587576+0000 mgr.smithi097.swvonl (mgr.14180) 2408 : cluster [DBG] pgmap v1700: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:30.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:30 smithi193 bash[28559]: cephadm 2023-11-17T07:42:28.607523+0000 mgr.smithi097.swvonl (mgr.14180) 2409 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.gkfysp on smithi193 2023-11-17T07:42:30.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:30 smithi193 bash[28559]: cluster 2023-11-17T07:42:29.184019+0000 mon.smithi097 (mon.0) 1287 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:42:30.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:30 smithi193 bash[28559]: cluster 2023-11-17T07:42:29.184094+0000 mon.smithi097 (mon.0) 1288 : cluster [INF] Cluster is now healthy 2023-11-17T07:42:31.422 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:31.422 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:40:19.979436Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.knxzrs on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:24.495507Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.hczihz on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-hczihz\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.hczihz\nDeploy daemon haproxy.nfs.foo.smithi193.hczihz ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.265073Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.mqptos on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-mqptos\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.mqptos\nDeploy daemon haproxy.nfs.foo.smithi097.mqptos ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:41:28.267526Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.xbuxqu on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:32.306 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:32.306 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:32 smithi193 bash[28559]: cluster 2023-11-17T07:42:30.588689+0000 mgr.smithi097.swvonl (mgr.14180) 2410 : cluster [DBG] pgmap v1701: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:32.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:32 smithi097 bash[24958]: cluster 2023-11-17T07:42:30.588689+0000 mgr.smithi097.swvonl (mgr.14180) 2410 : cluster [DBG] pgmap v1701: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:33.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:33 smithi097 bash[24958]: audit 2023-11-17T07:42:31.410826+0000 mgr.smithi097.swvonl (mgr.14180) 2411 : audit [DBG] from='client.15644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:33.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:33 smithi097 bash[24958]: audit 2023-11-17T07:42:33.024084+0000 mon.smithi097 (mon.0) 1289 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.gkfysp"}]: dispatch 2023-11-17T07:42:33.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:33.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:33 smithi193 bash[28559]: audit 2023-11-17T07:42:31.410826+0000 mgr.smithi097.swvonl (mgr.14180) 2411 : audit [DBG] from='client.15644 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:33.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:33 smithi193 bash[28559]: audit 2023-11-17T07:42:33.024084+0000 mon.smithi097 (mon.0) 1289 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.gkfysp"}]: dispatch 2023-11-17T07:42:34.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: cluster 2023-11-17T07:42:32.589413+0000 mgr.smithi097.swvonl (mgr.14180) 2412 : cluster [DBG] pgmap v1702: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: cephadm 2023-11-17T07:42:33.022944+0000 mgr.smithi097.swvonl (mgr.14180) 2413 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: cephadm 2023-11-17T07:42:33.023439+0000 mgr.smithi097.swvonl (mgr.14180) 2414 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: cephadm 2023-11-17T07:42:33.025855+0000 mgr.smithi097.swvonl (mgr.14180) 2415 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:34 smithi097 bash[24958]: cephadm 2023-11-17T07:42:33.030526+0000 mgr.smithi097.swvonl (mgr.14180) 2416 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lcmzhn on smithi097 2023-11-17T07:42:34.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: cluster 2023-11-17T07:42:32.589413+0000 mgr.smithi097.swvonl (mgr.14180) 2412 : cluster [DBG] pgmap v1702: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: cephadm 2023-11-17T07:42:33.022944+0000 mgr.smithi097.swvonl (mgr.14180) 2413 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: cephadm 2023-11-17T07:42:33.023439+0000 mgr.smithi097.swvonl (mgr.14180) 2414 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: cephadm 2023-11-17T07:42:33.025855+0000 mgr.smithi097.swvonl (mgr.14180) 2415 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.gkfysp ... 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:34.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:34.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:34.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:34 smithi193 bash[28559]: cephadm 2023-11-17T07:42:33.030526+0000 mgr.smithi097.swvonl (mgr.14180) 2416 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.lcmzhn on smithi097 2023-11-17T07:42:36.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:36 smithi097 bash[24958]: cluster 2023-11-17T07:42:34.590370+0000 mgr.smithi097.swvonl (mgr.14180) 2417 : cluster [DBG] pgmap v1703: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:36.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:36 smithi193 bash[28559]: cluster 2023-11-17T07:42:34.590370+0000 mgr.smithi097.swvonl (mgr.14180) 2417 : cluster [DBG] pgmap v1703: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:37.318 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:37.318 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:37.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:37 smithi097 bash[24958]: audit 2023-11-17T07:42:36.856510+0000 mon.smithi097 (mon.0) 1290 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lcmzhn"}]: dispatch 2023-11-17T07:42:37.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:37 smithi193 bash[28559]: audit 2023-11-17T07:42:36.856510+0000 mon.smithi097 (mon.0) 1290 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.lcmzhn"}]: dispatch 2023-11-17T07:42:38.272 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cluster 2023-11-17T07:42:36.591371+0000 mgr.smithi097.swvonl (mgr.14180) 2418 : cluster [DBG] pgmap v1704: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cephadm 2023-11-17T07:42:36.855706+0000 mgr.smithi097.swvonl (mgr.14180) 2419 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cephadm 2023-11-17T07:42:36.856024+0000 mgr.smithi097.swvonl (mgr.14180) 2420 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cephadm 2023-11-17T07:42:36.857629+0000 mgr.smithi097.swvonl (mgr.14180) 2421 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cephadm 2023-11-17T07:42:36.860622+0000 mgr.smithi097.swvonl (mgr.14180) 2422 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cephadm 2023-11-17T07:42:36.863155+0000 mgr.smithi097.swvonl (mgr.14180) 2423 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cluster 2023-11-17T07:42:36.864336+0000 mgr.smithi097.swvonl (mgr.14180) 2424 : cluster [DBG] pgmap v1705: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 B/s rd, 0 op/s 2023-11-17T07:42:38.554 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:38 smithi097 bash[24958]: cluster 2023-11-17T07:42:37.228015+0000 mon.smithi097 (mon.0) 1291 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:42:38.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cluster 2023-11-17T07:42:36.591371+0000 mgr.smithi097.swvonl (mgr.14180) 2418 : cluster [DBG] pgmap v1704: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cephadm 2023-11-17T07:42:36.855706+0000 mgr.smithi097.swvonl (mgr.14180) 2419 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cephadm 2023-11-17T07:42:36.856024+0000 mgr.smithi097.swvonl (mgr.14180) 2420 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cephadm 2023-11-17T07:42:36.857629+0000 mgr.smithi097.swvonl (mgr.14180) 2421 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.lcmzhn ... 2023-11-17T07:42:38.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cephadm 2023-11-17T07:42:36.860622+0000 mgr.smithi097.swvonl (mgr.14180) 2422 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cephadm 2023-11-17T07:42:36.863155+0000 mgr.smithi097.swvonl (mgr.14180) 2423 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:42:38.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cluster 2023-11-17T07:42:36.864336+0000 mgr.smithi097.swvonl (mgr.14180) 2424 : cluster [DBG] pgmap v1705: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 B/s rd, 0 op/s 2023-11-17T07:42:38.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:38 smithi193 bash[28559]: cluster 2023-11-17T07:42:37.228015+0000 mon.smithi097 (mon.0) 1291 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:42:39.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:39.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:39 smithi097 bash[24958]: audit 2023-11-17T07:42:37.306920+0000 mgr.smithi097.swvonl (mgr.14180) 2425 : audit [DBG] from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:39 smithi193 bash[28559]: audit 2023-11-17T07:42:37.306920+0000 mgr.smithi097.swvonl (mgr.14180) 2425 : audit [DBG] from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:40.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:40 smithi097 bash[24958]: cluster 2023-11-17T07:42:38.865369+0000 mgr.smithi097.swvonl (mgr.14180) 2426 : cluster [DBG] pgmap v1706: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:40.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:40 smithi193 bash[28559]: cluster 2023-11-17T07:42:38.865369+0000 mgr.smithi097.swvonl (mgr.14180) 2426 : cluster [DBG] pgmap v1706: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:41.742 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:41 smithi097 bash[24958]: audit 2023-11-17T07:42:40.368693+0000 mon.smithi097 (mon.0) 1292 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:41.742 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:41 smithi097 bash[24958]: cluster 2023-11-17T07:42:40.866415+0000 mgr.smithi097.swvonl (mgr.14180) 2427 : cluster [DBG] pgmap v1707: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:41.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:41 smithi193 bash[28559]: audit 2023-11-17T07:42:40.368693+0000 mon.smithi097 (mon.0) 1292 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:42:41.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:41 smithi193 bash[28559]: cluster 2023-11-17T07:42:40.866415+0000 mgr.smithi097.swvonl (mgr.14180) 2427 : cluster [DBG] pgmap v1707: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:43.228 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:43.228 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:44.217 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:44.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:44 smithi097 bash[24958]: cluster 2023-11-17T07:42:42.866946+0000 mgr.smithi097.swvonl (mgr.14180) 2428 : cluster [DBG] pgmap v1708: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:44.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:44 smithi097 bash[24958]: audit 2023-11-17T07:42:43.221220+0000 mgr.smithi097.swvonl (mgr.14180) 2429 : audit [DBG] from='client.15652 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:44.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:44 smithi193 bash[28559]: cluster 2023-11-17T07:42:42.866946+0000 mgr.smithi097.swvonl (mgr.14180) 2428 : cluster [DBG] pgmap v1708: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:44.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:44 smithi193 bash[28559]: audit 2023-11-17T07:42:43.221220+0000 mgr.smithi097.swvonl (mgr.14180) 2429 : audit [DBG] from='client.15652 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:45.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:46.460 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:46 smithi097 bash[24958]: cluster 2023-11-17T07:42:44.868245+0000 mgr.smithi097.swvonl (mgr.14180) 2430 : cluster [DBG] pgmap v1709: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:46.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:46 smithi193 bash[28559]: cluster 2023-11-17T07:42:44.868245+0000 mgr.smithi097.swvonl (mgr.14180) 2430 : cluster [DBG] pgmap v1709: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:48.461 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:48 smithi097 bash[24958]: cluster 2023-11-17T07:42:46.869258+0000 mgr.smithi097.swvonl (mgr.14180) 2431 : cluster [DBG] pgmap v1710: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:48.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:48 smithi193 bash[28559]: cluster 2023-11-17T07:42:46.869258+0000 mgr.smithi097.swvonl (mgr.14180) 2431 : cluster [DBG] pgmap v1710: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:42:48.874 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:48.874 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:49.735 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:50.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:50 smithi097 bash[24958]: audit 2023-11-17T07:42:48.862343+0000 mgr.smithi097.swvonl (mgr.14180) 2432 : audit [DBG] from='client.15656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:50.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:50 smithi097 bash[24958]: cluster 2023-11-17T07:42:48.870547+0000 mgr.smithi097.swvonl (mgr.14180) 2433 : cluster [DBG] pgmap v1711: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:50.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:50 smithi193 bash[28559]: audit 2023-11-17T07:42:48.862343+0000 mgr.smithi097.swvonl (mgr.14180) 2432 : audit [DBG] from='client.15656 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:50.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:50 smithi193 bash[28559]: cluster 2023-11-17T07:42:48.870547+0000 mgr.smithi097.swvonl (mgr.14180) 2433 : cluster [DBG] pgmap v1711: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:50.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:52.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:52 smithi097 bash[24958]: cluster 2023-11-17T07:42:50.871689+0000 mgr.smithi097.swvonl (mgr.14180) 2434 : cluster [DBG] pgmap v1712: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:52.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:52 smithi193 bash[28559]: cluster 2023-11-17T07:42:50.871689+0000 mgr.smithi097.swvonl (mgr.14180) 2434 : cluster [DBG] pgmap v1712: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:42:54.529 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:42:54.529 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:42:54.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:54 smithi097 bash[24958]: cluster 2023-11-17T07:42:52.872692+0000 mgr.smithi097.swvonl (mgr.14180) 2435 : cluster [DBG] pgmap v1713: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:54.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:54 smithi193 bash[28559]: cluster 2023-11-17T07:42:52.872692+0000 mgr.smithi097.swvonl (mgr.14180) 2435 : cluster [DBG] pgmap v1713: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:55.471 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:42:56.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:42:56.486 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:56 smithi097 bash[24958]: audit 2023-11-17T07:42:54.516529+0000 mgr.smithi097.swvonl (mgr.14180) 2436 : audit [DBG] from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:56.486 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:56 smithi097 bash[24958]: cluster 2023-11-17T07:42:54.873404+0000 mgr.smithi097.swvonl (mgr.14180) 2437 : cluster [DBG] pgmap v1714: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:56.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:56 smithi193 bash[28559]: audit 2023-11-17T07:42:54.516529+0000 mgr.smithi097.swvonl (mgr.14180) 2436 : audit [DBG] from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:42:56.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:56 smithi193 bash[28559]: cluster 2023-11-17T07:42:54.873404+0000 mgr.smithi097.swvonl (mgr.14180) 2437 : cluster [DBG] pgmap v1714: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:42:58 smithi097 bash[24958]: cluster 2023-11-17T07:42:56.873947+0000 mgr.smithi097.swvonl (mgr.14180) 2438 : cluster [DBG] pgmap v1715: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:42:58.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:42:58 smithi193 bash[28559]: cluster 2023-11-17T07:42:56.873947+0000 mgr.smithi097.swvonl (mgr.14180) 2438 : cluster [DBG] pgmap v1715: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:00.361 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:00.361 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:00.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:00 smithi097 bash[24958]: cluster 2023-11-17T07:42:58.874839+0000 mgr.smithi097.swvonl (mgr.14180) 2439 : cluster [DBG] pgmap v1716: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:00.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:00 smithi193 bash[28559]: cluster 2023-11-17T07:42:58.874839+0000 mgr.smithi097.swvonl (mgr.14180) 2439 : cluster [DBG] pgmap v1716: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:01.346 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:02.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:43:02.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:02 smithi097 bash[24958]: audit 2023-11-17T07:43:00.352097+0000 mgr.smithi097.swvonl (mgr.14180) 2440 : audit [DBG] from='client.15664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:02.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:02 smithi097 bash[24958]: cluster 2023-11-17T07:43:00.876219+0000 mgr.smithi097.swvonl (mgr.14180) 2441 : cluster [DBG] pgmap v1717: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:02 smithi193 bash[28559]: audit 2023-11-17T07:43:00.352097+0000 mgr.smithi097.swvonl (mgr.14180) 2440 : audit [DBG] from='client.15664 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:02 smithi193 bash[28559]: cluster 2023-11-17T07:43:00.876219+0000 mgr.smithi097.swvonl (mgr.14180) 2441 : cluster [DBG] pgmap v1717: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:04.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:04 smithi097 bash[24958]: cluster 2023-11-17T07:43:02.877013+0000 mgr.smithi097.swvonl (mgr.14180) 2442 : cluster [DBG] pgmap v1718: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:04.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:04 smithi193 bash[28559]: cluster 2023-11-17T07:43:02.877013+0000 mgr.smithi097.swvonl (mgr.14180) 2442 : cluster [DBG] pgmap v1718: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:05.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:05.966 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:06.270 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:06 smithi097 bash[24958]: cluster 2023-11-17T07:43:04.878342+0000 mgr.smithi097.swvonl (mgr.14180) 2443 : cluster [DBG] pgmap v1719: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:06.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:06 smithi193 bash[28559]: cluster 2023-11-17T07:43:04.878342+0000 mgr.smithi097.swvonl (mgr.14180) 2443 : cluster [DBG] pgmap v1719: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:06.891 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:07.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:07 smithi097 bash[24958]: audit 2023-11-17T07:43:05.959701+0000 mgr.smithi097.swvonl (mgr.14180) 2444 : audit [DBG] from='client.15668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:07.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:07 smithi193 bash[28559]: audit 2023-11-17T07:43:05.959701+0000 mgr.smithi097.swvonl (mgr.14180) 2444 : audit [DBG] from='client.15668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:07.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:43:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:08 smithi097 bash[24958]: cluster 2023-11-17T07:43:06.879545+0000 mgr.smithi097.swvonl (mgr.14180) 2445 : cluster [DBG] pgmap v1720: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:08.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:08 smithi193 bash[28559]: cluster 2023-11-17T07:43:06.879545+0000 mgr.smithi097.swvonl (mgr.14180) 2445 : cluster [DBG] pgmap v1720: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:10 smithi097 bash[24958]: cluster 2023-11-17T07:43:08.880765+0000 mgr.smithi097.swvonl (mgr.14180) 2446 : cluster [DBG] pgmap v1721: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:10 smithi193 bash[28559]: cluster 2023-11-17T07:43:08.880765+0000 mgr.smithi097.swvonl (mgr.14180) 2446 : cluster [DBG] pgmap v1721: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:12.130 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:12.130 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:12.468 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:12 smithi097 bash[24958]: cluster 2023-11-17T07:43:10.882380+0000 mgr.smithi097.swvonl (mgr.14180) 2447 : cluster [DBG] pgmap v1722: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:12.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:12 smithi193 bash[28559]: cluster 2023-11-17T07:43:10.882380+0000 mgr.smithi097.swvonl (mgr.14180) 2447 : cluster [DBG] pgmap v1722: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:13.084 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:13.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:13 smithi097 bash[24958]: audit 2023-11-17T07:43:12.116761+0000 mgr.smithi097.swvonl (mgr.14180) 2448 : audit [DBG] from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:13.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:13 smithi193 bash[28559]: audit 2023-11-17T07:43:12.116761+0000 mgr.smithi097.swvonl (mgr.14180) 2448 : audit [DBG] from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:14.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:43:14.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:14 smithi097 bash[24958]: cluster 2023-11-17T07:43:12.883550+0000 mgr.smithi097.swvonl (mgr.14180) 2449 : cluster [DBG] pgmap v1723: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:14.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:14 smithi193 bash[28559]: cluster 2023-11-17T07:43:12.883550+0000 mgr.smithi097.swvonl (mgr.14180) 2449 : cluster [DBG] pgmap v1723: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:16.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:16 smithi097 bash[24958]: cluster 2023-11-17T07:43:14.884509+0000 mgr.smithi097.swvonl (mgr.14180) 2450 : cluster [DBG] pgmap v1724: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:16.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:16 smithi193 bash[28559]: cluster 2023-11-17T07:43:14.884509+0000 mgr.smithi097.swvonl (mgr.14180) 2450 : cluster [DBG] pgmap v1724: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:18.005 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:18.005 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:18.293 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:18 smithi097 bash[24958]: cluster 2023-11-17T07:43:16.885083+0000 mgr.smithi097.swvonl (mgr.14180) 2451 : cluster [DBG] pgmap v1725: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:18.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:18 smithi193 bash[28559]: cluster 2023-11-17T07:43:16.885083+0000 mgr.smithi097.swvonl (mgr.14180) 2451 : cluster [DBG] pgmap v1725: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:18.906 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:19.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:19 smithi097 bash[24958]: audit 2023-11-17T07:43:17.993294+0000 mgr.smithi097.swvonl (mgr.14180) 2452 : audit [DBG] from='client.15676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:19 smithi193 bash[28559]: audit 2023-11-17T07:43:17.993294+0000 mgr.smithi097.swvonl (mgr.14180) 2452 : audit [DBG] from='client.15676 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:19.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:43:20.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:20 smithi097 bash[24958]: cluster 2023-11-17T07:43:18.886041+0000 mgr.smithi097.swvonl (mgr.14180) 2453 : cluster [DBG] pgmap v1726: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:20.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:20 smithi193 bash[28559]: cluster 2023-11-17T07:43:18.886041+0000 mgr.smithi097.swvonl (mgr.14180) 2453 : cluster [DBG] pgmap v1726: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:22 smithi097 bash[24958]: cluster 2023-11-17T07:43:20.887385+0000 mgr.smithi097.swvonl (mgr.14180) 2454 : cluster [DBG] pgmap v1727: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:22.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:22 smithi193 bash[28559]: cluster 2023-11-17T07:43:20.887385+0000 mgr.smithi097.swvonl (mgr.14180) 2454 : cluster [DBG] pgmap v1727: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:43:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:24.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:24 smithi097 bash[24958]: cluster 2023-11-17T07:43:22.888353+0000 mgr.smithi097.swvonl (mgr.14180) 2455 : cluster [DBG] pgmap v1728: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:24.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:24 smithi193 bash[28559]: cluster 2023-11-17T07:43:22.888353+0000 mgr.smithi097.swvonl (mgr.14180) 2455 : cluster [DBG] pgmap v1728: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:24.812 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:25.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:25 smithi097 bash[24958]: audit 2023-11-17T07:43:23.876521+0000 mgr.smithi097.swvonl (mgr.14180) 2456 : audit [DBG] from='client.15680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:25 smithi193 bash[28559]: audit 2023-11-17T07:43:23.876521+0000 mgr.smithi097.swvonl (mgr.14180) 2456 : audit [DBG] from='client.15680 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:43:25.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:43:26.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:26 smithi097 bash[24958]: cluster 2023-11-17T07:43:24.889775+0000 mgr.smithi097.swvonl (mgr.14180) 2457 : cluster [DBG] pgmap v1729: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:26.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:26 smithi193 bash[28559]: cluster 2023-11-17T07:43:24.889775+0000 mgr.smithi097.swvonl (mgr.14180) 2457 : cluster [DBG] pgmap v1729: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:28.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:28 smithi097 bash[24958]: cluster 2023-11-17T07:43:26.890941+0000 mgr.smithi097.swvonl (mgr.14180) 2458 : cluster [DBG] pgmap v1730: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:28.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:28 smithi193 bash[28559]: cluster 2023-11-17T07:43:26.890941+0000 mgr.smithi097.swvonl (mgr.14180) 2458 : cluster [DBG] pgmap v1730: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:29.789 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:43:29.789 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:41:28.269194Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.lkjqhn on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:33.025605Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.gkfysp on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-gkfysp\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.gkfysp\nDeploy daemon haproxy.nfs.foo.smithi193.gkfysp ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.857477Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.lcmzhn on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-lcmzhn\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.lcmzhn\nDeploy daemon haproxy.nfs.foo.smithi097.lcmzhn ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:42:36.860423Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.nvlivo on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:42:36.862996Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eiiamj on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:43:30.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:30 smithi097 bash[24958]: cluster 2023-11-17T07:43:28.891794+0000 mgr.smithi097.swvonl (mgr.14180) 2459 : cluster [DBG] pgmap v1731: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:30.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:30 smithi193 bash[28559]: cluster 2023-11-17T07:43:28.891794+0000 mgr.smithi097.swvonl (mgr.14180) 2459 : cluster [DBG] pgmap v1731: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:43:30.727 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:43:31.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:31 smithi097 bash[24958]: audit 2023-11-17T07:43:29.782556+0000 mgr.smithi097.swvonl (mgr.14180) 2460 : audit [DBG] from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:44:52.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:44:52.976 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:31 smithi193 bash[28559]: audit 2023-11-17T07:43:29.782556+0000 mgr.smithi097.swvonl (mgr.14180) 2460 : audit [DBG] from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:44:52.976 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:32 smithi193 bash[28559]: cluster 2023-11-17T07:43:30.893186+0000 mgr.smithi097.swvonl (mgr.14180) 2461 : cluster [DBG] pgmap v1732: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.976 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:34 smithi193 bash[28559]: cluster 2023-11-17T07:43:32.894135+0000 mgr.smithi097.swvonl (mgr.14180) 2462 : cluster [DBG] pgmap v1733: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.976 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:36 smithi193 bash[28559]: cluster 2023-11-17T07:43:34.895630+0000 mgr.smithi097.swvonl (mgr.14180) 2463 : cluster [DBG] pgmap v1734: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.976 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:36.868463+0000 mon.smithi097 (mon.0) 1293 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:37.172219+0000 mon.smithi097 (mon.0) 1294 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:37.173831+0000 mon.smithi097 (mon.0) 1295 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:37.182267+0000 mon.smithi097 (mon.0) 1296 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:37.190397+0000 mon.smithi097 (mon.0) 1297 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:37 smithi193 bash[28559]: audit 2023-11-17T07:43:37.202617+0000 mon.smithi097 (mon.0) 1298 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:38 smithi193 bash[28559]: cluster 2023-11-17T07:43:36.896577+0000 mgr.smithi097.swvonl (mgr.14180) 2464 : cluster [DBG] pgmap v1735: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.977 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:38 smithi193 bash[28559]: cluster 2023-11-17T07:43:37.183122+0000 mgr.smithi097.swvonl (mgr.14180) 2465 : cluster [DBG] pgmap v1736: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:52.978 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:38 smithi193 bash[28559]: cephadm 2023-11-17T07:43:37.207339+0000 mgr.smithi097.swvonl (mgr.14180) 2466 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.bsypvf on smithi193 2023-11-17T07:44:52.978 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:38 smithi193 bash[28559]: cluster 2023-11-17T07:43:37.247189+0000 mon.smithi097 (mon.0) 1299 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:44:52.980 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:38 smithi193 bash[28559]: cluster 2023-11-17T07:43:37.247253+0000 mon.smithi097 (mon.0) 1300 : cluster [INF] Cluster is now healthy 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:40 smithi193 bash[28559]: cluster 2023-11-17T07:43:39.183944+0000 mgr.smithi097.swvonl (mgr.14180) 2467 : cluster [DBG] pgmap v1737: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:41 smithi193 bash[28559]: audit 2023-11-17T07:43:41.154720+0000 mon.smithi097 (mon.0) 1301 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.bsypvf"}]: dispatch 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: cephadm 2023-11-17T07:43:41.153699+0000 mgr.smithi097.swvonl (mgr.14180) 2468 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.981 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.982 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:44:52.983 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.984 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: cephadm 2023-11-17T07:43:41.154128+0000 mgr.smithi097.swvonl (mgr.14180) 2469 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: cephadm 2023-11-17T07:43:41.156417+0000 mgr.smithi097.swvonl (mgr.14180) 2470 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.bsypvf on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.985 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.986 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: cephadm 2023-11-17T07:43:41.161103+0000 mgr.smithi097.swvonl (mgr.14180) 2471 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.zzfnhy on smithi097 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: cluster 2023-11-17T07:43:41.185035+0000 mgr.smithi097.swvonl (mgr.14180) 2472 : cluster [DBG] pgmap v1738: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:52.987 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:42 smithi193 bash[28559]: audit 2023-11-17T07:43:41.586883+0000 mon.smithi097 (mon.0) 1302 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:43 smithi193 bash[28559]: cluster 2023-11-17T07:43:43.185793+0000 mgr.smithi097.swvonl (mgr.14180) 2473 : cluster [DBG] pgmap v1739: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:44 smithi193 bash[28559]: audit 2023-11-17T07:43:44.904578+0000 mon.smithi097 (mon.0) 1303 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.zzfnhy"}]: dispatch 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: cephadm 2023-11-17T07:43:44.903515+0000 mgr.smithi097.swvonl (mgr.14180) 2474 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:52.988 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:45 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.989 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cephadm 2023-11-17T07:43:44.904003+0000 mgr.smithi097.swvonl (mgr.14180) 2475 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cephadm 2023-11-17T07:43:44.905922+0000 mgr.smithi097.swvonl (mgr.14180) 2476 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.zzfnhy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:52.990 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cephadm 2023-11-17T07:43:44.909660+0000 mgr.smithi097.swvonl (mgr.14180) 2477 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.qayaow on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cephadm 2023-11-17T07:43:44.912847+0000 mgr.smithi097.swvonl (mgr.14180) 2478 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cluster 2023-11-17T07:43:44.914298+0000 mgr.smithi097.swvonl (mgr.14180) 2479 : cluster [DBG] pgmap v1740: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: cluster 2023-11-17T07:43:44.965768+0000 mon.smithi097 (mon.0) 1304 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:46 smithi193 bash[28559]: audit 2023-11-17T07:43:45.418886+0000 mon.smithi097 (mon.0) 1305 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.991 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:48 smithi193 bash[28559]: cluster 2023-11-17T07:43:46.915314+0000 mgr.smithi097.swvonl (mgr.14180) 2480 : cluster [DBG] pgmap v1741: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:50 smithi193 bash[28559]: cluster 2023-11-17T07:43:48.916375+0000 mgr.smithi097.swvonl (mgr.14180) 2481 : cluster [DBG] pgmap v1742: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:52 smithi193 bash[28559]: cluster 2023-11-17T07:43:50.917697+0000 mgr.smithi097.swvonl (mgr.14180) 2482 : cluster [DBG] pgmap v1743: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 349 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:54 smithi193 bash[28559]: cluster 2023-11-17T07:43:52.918701+0000 mgr.smithi097.swvonl (mgr.14180) 2483 : cluster [DBG] pgmap v1744: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:56 smithi193 bash[28559]: cluster 2023-11-17T07:43:54.920186+0000 mgr.smithi097.swvonl (mgr.14180) 2484 : cluster [DBG] pgmap v1745: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:57 smithi193 bash[28559]: audit 2023-11-17T07:43:56.585912+0000 mon.smithi097 (mon.0) 1306 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:57 smithi193 bash[28559]: cluster 2023-11-17T07:43:56.921227+0000 mgr.smithi097.swvonl (mgr.14180) 2485 : cluster [DBG] pgmap v1746: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:43:59 smithi193 bash[28559]: cluster 2023-11-17T07:43:58.922323+0000 mgr.smithi097.swvonl (mgr.14180) 2486 : cluster [DBG] pgmap v1747: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:01 smithi193 bash[28559]: cluster 2023-11-17T07:44:00.923592+0000 mgr.smithi097.swvonl (mgr.14180) 2487 : cluster [DBG] pgmap v1748: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:04 smithi193 bash[28559]: cluster 2023-11-17T07:44:02.924567+0000 mgr.smithi097.swvonl (mgr.14180) 2488 : cluster [DBG] pgmap v1749: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:06 smithi193 bash[28559]: cluster 2023-11-17T07:44:04.925759+0000 mgr.smithi097.swvonl (mgr.14180) 2489 : cluster [DBG] pgmap v1750: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:08 smithi193 bash[28559]: cluster 2023-11-17T07:44:06.926802+0000 mgr.smithi097.swvonl (mgr.14180) 2490 : cluster [DBG] pgmap v1751: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:10 smithi193 bash[28559]: cluster 2023-11-17T07:44:08.927881+0000 mgr.smithi097.swvonl (mgr.14180) 2491 : cluster [DBG] pgmap v1752: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.992 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:12 smithi193 bash[28559]: cluster 2023-11-17T07:44:10.929265+0000 mgr.smithi097.swvonl (mgr.14180) 2492 : cluster [DBG] pgmap v1753: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:14 smithi193 bash[28559]: cluster 2023-11-17T07:44:12.930227+0000 mgr.smithi097.swvonl (mgr.14180) 2493 : cluster [DBG] pgmap v1754: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:16 smithi193 bash[28559]: cluster 2023-11-17T07:44:14.931547+0000 mgr.smithi097.swvonl (mgr.14180) 2494 : cluster [DBG] pgmap v1755: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:18 smithi193 bash[28559]: cluster 2023-11-17T07:44:16.932597+0000 mgr.smithi097.swvonl (mgr.14180) 2495 : cluster [DBG] pgmap v1756: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:20 smithi193 bash[28559]: cluster 2023-11-17T07:44:18.933671+0000 mgr.smithi097.swvonl (mgr.14180) 2496 : cluster [DBG] pgmap v1757: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:22 smithi193 bash[28559]: cluster 2023-11-17T07:44:20.934994+0000 mgr.smithi097.swvonl (mgr.14180) 2497 : cluster [DBG] pgmap v1758: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:24 smithi193 bash[28559]: cluster 2023-11-17T07:44:22.935741+0000 mgr.smithi097.swvonl (mgr.14180) 2498 : cluster [DBG] pgmap v1759: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:26 smithi193 bash[28559]: cluster 2023-11-17T07:44:24.937044+0000 mgr.smithi097.swvonl (mgr.14180) 2499 : cluster [DBG] pgmap v1760: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:28 smithi193 bash[28559]: cluster 2023-11-17T07:44:26.938026+0000 mgr.smithi097.swvonl (mgr.14180) 2500 : cluster [DBG] pgmap v1761: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:30 smithi193 bash[28559]: cluster 2023-11-17T07:44:28.939151+0000 mgr.smithi097.swvonl (mgr.14180) 2501 : cluster [DBG] pgmap v1762: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:32 smithi193 bash[28559]: cluster 2023-11-17T07:44:30.940282+0000 mgr.smithi097.swvonl (mgr.14180) 2502 : cluster [DBG] pgmap v1763: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:34 smithi193 bash[28559]: cluster 2023-11-17T07:44:32.941189+0000 mgr.smithi097.swvonl (mgr.14180) 2503 : cluster [DBG] pgmap v1764: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:35 smithi193 bash[28559]: cluster 2023-11-17T07:44:34.942612+0000 mgr.smithi097.swvonl (mgr.14180) 2504 : cluster [DBG] pgmap v1765: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.993 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:38 smithi193 bash[28559]: cluster 2023-11-17T07:44:36.943620+0000 mgr.smithi097.swvonl (mgr.14180) 2505 : cluster [DBG] pgmap v1766: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:40 smithi193 bash[28559]: cluster 2023-11-17T07:44:38.944674+0000 mgr.smithi097.swvonl (mgr.14180) 2506 : cluster [DBG] pgmap v1767: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:42 smithi193 bash[28559]: cluster 2023-11-17T07:44:40.945884+0000 mgr.smithi097.swvonl (mgr.14180) 2507 : cluster [DBG] pgmap v1768: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:44 smithi193 bash[28559]: cluster 2023-11-17T07:44:42.946847+0000 mgr.smithi097.swvonl (mgr.14180) 2508 : cluster [DBG] pgmap v1769: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:45 smithi193 bash[28559]: audit 2023-11-17T07:44:44.917689+0000 mon.smithi097 (mon.0) 1307 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: cluster 2023-11-17T07:44:44.947956+0000 mgr.smithi097.swvonl (mgr.14180) 2509 : cluster [DBG] pgmap v1770: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: audit 2023-11-17T07:44:45.220430+0000 mon.smithi097 (mon.0) 1308 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: audit 2023-11-17T07:44:45.224702+0000 mon.smithi097 (mon.0) 1309 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: audit 2023-11-17T07:44:45.232629+0000 mon.smithi097 (mon.0) 1310 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: cluster 2023-11-17T07:44:45.233503+0000 mgr.smithi097.swvonl (mgr.14180) 2510 : cluster [DBG] pgmap v1771: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: audit 2023-11-17T07:44:45.240610+0000 mon.smithi097 (mon.0) 1311 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: audit 2023-11-17T07:44:45.251713+0000 mon.smithi097 (mon.0) 1312 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.994 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:46 smithi193 bash[28559]: cephadm 2023-11-17T07:44:45.255080+0000 mgr.smithi097.swvonl (mgr.14180) 2511 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.iyjktt on smithi193 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:47 smithi193 bash[28559]: cluster 2023-11-17T07:44:46.234894+0000 mon.smithi097 (mon.0) 1313 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:47 smithi193 bash[28559]: cluster 2023-11-17T07:44:46.234956+0000 mon.smithi097 (mon.0) 1314 : cluster [INF] Cluster is now healthy 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:48 smithi193 bash[28559]: cluster 2023-11-17T07:44:47.234192+0000 mgr.smithi097.swvonl (mgr.14180) 2512 : cluster [DBG] pgmap v1772: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:49 smithi193 bash[28559]: audit 2023-11-17T07:44:49.218464+0000 mon.smithi097 (mon.0) 1315 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.iyjktt"}]: dispatch 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: cephadm 2023-11-17T07:44:49.217234+0000 mgr.smithi097.swvonl (mgr.14180) 2513 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.995 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.996 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: cephadm 2023-11-17T07:44:49.217732+0000 mgr.smithi097.swvonl (mgr.14180) 2514 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: cephadm 2023-11-17T07:44:49.220246+0000 mgr.smithi097.swvonl (mgr.14180) 2515 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.997 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: cephadm 2023-11-17T07:44:49.225001+0000 mgr.smithi097.swvonl (mgr.14180) 2516 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.fninro on smithi097 2023-11-17T07:44:52.998 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:50 smithi193 bash[28559]: cluster 2023-11-17T07:44:49.234731+0000 mgr.smithi097.swvonl (mgr.14180) 2517 : cluster [DBG] pgmap v1773: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:52 smithi193 bash[28559]: cluster 2023-11-17T07:44:51.235419+0000 mgr.smithi097.swvonl (mgr.14180) 2518 : cluster [DBG] pgmap v1774: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:32 smithi097 bash[24958]: cluster 2023-11-17T07:43:30.893186+0000 mgr.smithi097.swvonl (mgr.14180) 2461 : cluster [DBG] pgmap v1732: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:34 smithi097 bash[24958]: cluster 2023-11-17T07:43:32.894135+0000 mgr.smithi097.swvonl (mgr.14180) 2462 : cluster [DBG] pgmap v1733: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:36 smithi097 bash[24958]: cluster 2023-11-17T07:43:34.895630+0000 mgr.smithi097.swvonl (mgr.14180) 2463 : cluster [DBG] pgmap v1734: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:36.868463+0000 mon.smithi097 (mon.0) 1293 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:37.172219+0000 mon.smithi097 (mon.0) 1294 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:37.173831+0000 mon.smithi097 (mon.0) 1295 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:37.182267+0000 mon.smithi097 (mon.0) 1296 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:37.190397+0000 mon.smithi097 (mon.0) 1297 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:44:52.999 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:37 smithi097 bash[24958]: audit 2023-11-17T07:43:37.202617+0000 mon.smithi097 (mon.0) 1298 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:38 smithi097 bash[24958]: cluster 2023-11-17T07:43:36.896577+0000 mgr.smithi097.swvonl (mgr.14180) 2464 : cluster [DBG] pgmap v1735: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:38 smithi097 bash[24958]: cluster 2023-11-17T07:43:37.183122+0000 mgr.smithi097.swvonl (mgr.14180) 2465 : cluster [DBG] pgmap v1736: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:38 smithi097 bash[24958]: cephadm 2023-11-17T07:43:37.207339+0000 mgr.smithi097.swvonl (mgr.14180) 2466 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.bsypvf on smithi193 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:38 smithi097 bash[24958]: cluster 2023-11-17T07:43:37.247189+0000 mon.smithi097 (mon.0) 1299 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:38 smithi097 bash[24958]: cluster 2023-11-17T07:43:37.247253+0000 mon.smithi097 (mon.0) 1300 : cluster [INF] Cluster is now healthy 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:40 smithi097 bash[24958]: cluster 2023-11-17T07:43:39.183944+0000 mgr.smithi097.swvonl (mgr.14180) 2467 : cluster [DBG] pgmap v1737: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:41 smithi097 bash[24958]: audit 2023-11-17T07:43:41.154720+0000 mon.smithi097 (mon.0) 1301 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.bsypvf"}]: dispatch 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: cephadm 2023-11-17T07:43:41.153699+0000 mgr.smithi097.swvonl (mgr.14180) 2468 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.000 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.001 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.002 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: cephadm 2023-11-17T07:43:41.154128+0000 mgr.smithi097.swvonl (mgr.14180) 2469 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: cephadm 2023-11-17T07:43:41.156417+0000 mgr.smithi097.swvonl (mgr.14180) 2470 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.bsypvf on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-bsypvf 2023-11-17T07:44:53.003 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.bsypvf 2023-11-17T07:44:53.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.bsypvf ... 2023-11-17T07:44:53.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.004 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: cephadm 2023-11-17T07:43:41.161103+0000 mgr.smithi097.swvonl (mgr.14180) 2471 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.zzfnhy on smithi097 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: cluster 2023-11-17T07:43:41.185035+0000 mgr.smithi097.swvonl (mgr.14180) 2472 : cluster [DBG] pgmap v1738: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:42 smithi097 bash[24958]: audit 2023-11-17T07:43:41.586883+0000 mon.smithi097 (mon.0) 1302 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.005 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:43 smithi097 bash[24958]: cluster 2023-11-17T07:43:43.185793+0000 mgr.smithi097.swvonl (mgr.14180) 2473 : cluster [DBG] pgmap v1739: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:44 smithi097 bash[24958]: audit 2023-11-17T07:43:44.904578+0000 mon.smithi097 (mon.0) 1303 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.zzfnhy"}]: dispatch 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cephadm 2023-11-17T07:43:44.903515+0000 mgr.smithi097.swvonl (mgr.14180) 2474 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.006 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:44:53.007 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.008 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.009 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cephadm 2023-11-17T07:43:44.904003+0000 mgr.smithi097.swvonl (mgr.14180) 2475 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cephadm 2023-11-17T07:43:44.905922+0000 mgr.smithi097.swvonl (mgr.14180) 2476 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.zzfnhy on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-zzfnhy 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.zzfnhy 2023-11-17T07:44:53.010 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.zzfnhy ... 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cephadm 2023-11-17T07:43:44.909660+0000 mgr.smithi097.swvonl (mgr.14180) 2477 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.qayaow on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cephadm 2023-11-17T07:43:44.912847+0000 mgr.smithi097.swvonl (mgr.14180) 2478 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cluster 2023-11-17T07:43:44.914298+0000 mgr.smithi097.swvonl (mgr.14180) 2479 : cluster [DBG] pgmap v1740: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: cluster 2023-11-17T07:43:44.965768+0000 mon.smithi097 (mon.0) 1304 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:44:53.011 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:45 smithi097 bash[24958]: audit 2023-11-17T07:43:45.418886+0000 mon.smithi097 (mon.0) 1305 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:48 smithi097 bash[24958]: cluster 2023-11-17T07:43:46.915314+0000 mgr.smithi097.swvonl (mgr.14180) 2480 : cluster [DBG] pgmap v1741: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:50 smithi097 bash[24958]: cluster 2023-11-17T07:43:48.916375+0000 mgr.smithi097.swvonl (mgr.14180) 2481 : cluster [DBG] pgmap v1742: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:52 smithi097 bash[24958]: cluster 2023-11-17T07:43:50.917697+0000 mgr.smithi097.swvonl (mgr.14180) 2482 : cluster [DBG] pgmap v1743: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 349 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:54 smithi097 bash[24958]: cluster 2023-11-17T07:43:52.918701+0000 mgr.smithi097.swvonl (mgr.14180) 2483 : cluster [DBG] pgmap v1744: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:56 smithi097 bash[24958]: cluster 2023-11-17T07:43:54.920186+0000 mgr.smithi097.swvonl (mgr.14180) 2484 : cluster [DBG] pgmap v1745: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:57 smithi097 bash[24958]: audit 2023-11-17T07:43:56.585912+0000 mon.smithi097 (mon.0) 1306 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:57 smithi097 bash[24958]: cluster 2023-11-17T07:43:56.921227+0000 mgr.smithi097.swvonl (mgr.14180) 2485 : cluster [DBG] pgmap v1746: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:43:59 smithi097 bash[24958]: cluster 2023-11-17T07:43:58.922323+0000 mgr.smithi097.swvonl (mgr.14180) 2486 : cluster [DBG] pgmap v1747: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:01 smithi097 bash[24958]: cluster 2023-11-17T07:44:00.923592+0000 mgr.smithi097.swvonl (mgr.14180) 2487 : cluster [DBG] pgmap v1748: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:04 smithi097 bash[24958]: cluster 2023-11-17T07:44:02.924567+0000 mgr.smithi097.swvonl (mgr.14180) 2488 : cluster [DBG] pgmap v1749: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:06 smithi097 bash[24958]: cluster 2023-11-17T07:44:04.925759+0000 mgr.smithi097.swvonl (mgr.14180) 2489 : cluster [DBG] pgmap v1750: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.012 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:08 smithi097 bash[24958]: cluster 2023-11-17T07:44:06.926802+0000 mgr.smithi097.swvonl (mgr.14180) 2490 : cluster [DBG] pgmap v1751: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:10 smithi097 bash[24958]: cluster 2023-11-17T07:44:08.927881+0000 mgr.smithi097.swvonl (mgr.14180) 2491 : cluster [DBG] pgmap v1752: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:12 smithi097 bash[24958]: cluster 2023-11-17T07:44:10.929265+0000 mgr.smithi097.swvonl (mgr.14180) 2492 : cluster [DBG] pgmap v1753: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:14 smithi097 bash[24958]: cluster 2023-11-17T07:44:12.930227+0000 mgr.smithi097.swvonl (mgr.14180) 2493 : cluster [DBG] pgmap v1754: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:16 smithi097 bash[24958]: cluster 2023-11-17T07:44:14.931547+0000 mgr.smithi097.swvonl (mgr.14180) 2494 : cluster [DBG] pgmap v1755: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:18 smithi097 bash[24958]: cluster 2023-11-17T07:44:16.932597+0000 mgr.smithi097.swvonl (mgr.14180) 2495 : cluster [DBG] pgmap v1756: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:20 smithi097 bash[24958]: cluster 2023-11-17T07:44:18.933671+0000 mgr.smithi097.swvonl (mgr.14180) 2496 : cluster [DBG] pgmap v1757: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:22 smithi097 bash[24958]: cluster 2023-11-17T07:44:20.934994+0000 mgr.smithi097.swvonl (mgr.14180) 2497 : cluster [DBG] pgmap v1758: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:24 smithi097 bash[24958]: cluster 2023-11-17T07:44:22.935741+0000 mgr.smithi097.swvonl (mgr.14180) 2498 : cluster [DBG] pgmap v1759: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:26 smithi097 bash[24958]: cluster 2023-11-17T07:44:24.937044+0000 mgr.smithi097.swvonl (mgr.14180) 2499 : cluster [DBG] pgmap v1760: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:28 smithi097 bash[24958]: cluster 2023-11-17T07:44:26.938026+0000 mgr.smithi097.swvonl (mgr.14180) 2500 : cluster [DBG] pgmap v1761: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:30 smithi097 bash[24958]: cluster 2023-11-17T07:44:28.939151+0000 mgr.smithi097.swvonl (mgr.14180) 2501 : cluster [DBG] pgmap v1762: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:32 smithi097 bash[24958]: cluster 2023-11-17T07:44:30.940282+0000 mgr.smithi097.swvonl (mgr.14180) 2502 : cluster [DBG] pgmap v1763: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:53.013 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:34 smithi097 bash[24958]: cluster 2023-11-17T07:44:32.941189+0000 mgr.smithi097.swvonl (mgr.14180) 2503 : cluster [DBG] pgmap v1764: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:35 smithi097 bash[24958]: cluster 2023-11-17T07:44:34.942612+0000 mgr.smithi097.swvonl (mgr.14180) 2504 : cluster [DBG] pgmap v1765: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:38 smithi097 bash[24958]: cluster 2023-11-17T07:44:36.943620+0000 mgr.smithi097.swvonl (mgr.14180) 2505 : cluster [DBG] pgmap v1766: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:40 smithi097 bash[24958]: cluster 2023-11-17T07:44:38.944674+0000 mgr.smithi097.swvonl (mgr.14180) 2506 : cluster [DBG] pgmap v1767: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:42 smithi097 bash[24958]: cluster 2023-11-17T07:44:40.945884+0000 mgr.smithi097.swvonl (mgr.14180) 2507 : cluster [DBG] pgmap v1768: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:44 smithi097 bash[24958]: cluster 2023-11-17T07:44:42.946847+0000 mgr.smithi097.swvonl (mgr.14180) 2508 : cluster [DBG] pgmap v1769: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:45 smithi097 bash[24958]: audit 2023-11-17T07:44:44.917689+0000 mon.smithi097 (mon.0) 1307 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: cluster 2023-11-17T07:44:44.947956+0000 mgr.smithi097.swvonl (mgr.14180) 2509 : cluster [DBG] pgmap v1770: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: audit 2023-11-17T07:44:45.220430+0000 mon.smithi097 (mon.0) 1308 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: audit 2023-11-17T07:44:45.224702+0000 mon.smithi097 (mon.0) 1309 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: audit 2023-11-17T07:44:45.232629+0000 mon.smithi097 (mon.0) 1310 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: cluster 2023-11-17T07:44:45.233503+0000 mgr.smithi097.swvonl (mgr.14180) 2510 : cluster [DBG] pgmap v1771: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:44:53.014 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: audit 2023-11-17T07:44:45.240610+0000 mon.smithi097 (mon.0) 1311 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: audit 2023-11-17T07:44:45.251713+0000 mon.smithi097 (mon.0) 1312 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:46 smithi097 bash[24958]: cephadm 2023-11-17T07:44:45.255080+0000 mgr.smithi097.swvonl (mgr.14180) 2511 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.iyjktt on smithi193 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:47 smithi097 bash[24958]: cluster 2023-11-17T07:44:46.234894+0000 mon.smithi097 (mon.0) 1313 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:47 smithi097 bash[24958]: cluster 2023-11-17T07:44:46.234956+0000 mon.smithi097 (mon.0) 1314 : cluster [INF] Cluster is now healthy 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:48 smithi097 bash[24958]: cluster 2023-11-17T07:44:47.234192+0000 mgr.smithi097.swvonl (mgr.14180) 2512 : cluster [DBG] pgmap v1772: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:49 smithi097 bash[24958]: audit 2023-11-17T07:44:49.218464+0000 mon.smithi097 (mon.0) 1315 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.iyjktt"}]: dispatch 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: cephadm 2023-11-17T07:44:49.217234+0000 mgr.smithi097.swvonl (mgr.14180) 2513 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:53.015 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:53.016 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.017 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: cephadm 2023-11-17T07:44:49.217732+0000 mgr.smithi097.swvonl (mgr.14180) 2514 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: cephadm 2023-11-17T07:44:49.220246+0000 mgr.smithi097.swvonl (mgr.14180) 2515 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.iyjktt ... 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:53.018 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:53.019 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: cephadm 2023-11-17T07:44:49.225001+0000 mgr.smithi097.swvonl (mgr.14180) 2516 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.fninro on smithi097 2023-11-17T07:44:53.019 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:50 smithi097 bash[24958]: cluster 2023-11-17T07:44:49.234731+0000 mgr.smithi097.swvonl (mgr.14180) 2517 : cluster [DBG] pgmap v1773: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:53.019 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:52 smithi097 bash[24958]: cluster 2023-11-17T07:44:51.235419+0000 mgr.smithi097.swvonl (mgr.14180) 2518 : cluster [DBG] pgmap v1774: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:44:53.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:53 smithi097 bash[24958]: audit 2023-11-17T07:44:52.961960+0000 mon.smithi097 (mon.0) 1316 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.fninro"}]: dispatch 2023-11-17T07:44:53.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:53 smithi193 bash[28559]: audit 2023-11-17T07:44:52.961960+0000 mon.smithi097 (mon.0) 1316 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.fninro"}]: dispatch 2023-11-17T07:44:54.526 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cephadm 2023-11-17T07:44:52.960747+0000 mgr.smithi097.swvonl (mgr.14180) 2519 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.528 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cephadm 2023-11-17T07:44:52.961285+0000 mgr.smithi097.swvonl (mgr.14180) 2520 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cephadm 2023-11-17T07:44:52.963540+0000 mgr.smithi097.swvonl (mgr.14180) 2521 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cephadm 2023-11-17T07:44:52.966784+0000 mgr.smithi097.swvonl (mgr.14180) 2522 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cephadm 2023-11-17T07:44:52.969465+0000 mgr.smithi097.swvonl (mgr.14180) 2523 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cluster 2023-11-17T07:44:52.970813+0000 mgr.smithi097.swvonl (mgr.14180) 2524 : cluster [DBG] pgmap v1775: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:54.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:54 smithi097 bash[24958]: cluster 2023-11-17T07:44:53.258467+0000 mon.smithi097 (mon.0) 1317 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:44:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cephadm 2023-11-17T07:44:52.960747+0000 mgr.smithi097.swvonl (mgr.14180) 2519 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cephadm 2023-11-17T07:44:52.961285+0000 mgr.smithi097.swvonl (mgr.14180) 2520 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cephadm 2023-11-17T07:44:52.963540+0000 mgr.smithi097.swvonl (mgr.14180) 2521 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.fninro ... 2023-11-17T07:44:54.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cephadm 2023-11-17T07:44:52.966784+0000 mgr.smithi097.swvonl (mgr.14180) 2522 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cephadm 2023-11-17T07:44:52.969465+0000 mgr.smithi097.swvonl (mgr.14180) 2523 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cluster 2023-11-17T07:44:52.970813+0000 mgr.smithi097.swvonl (mgr.14180) 2524 : cluster [DBG] pgmap v1775: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:54.561 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:54 smithi193 bash[28559]: cluster 2023-11-17T07:44:53.258467+0000 mon.smithi097 (mon.0) 1317 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:44:56.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:56 smithi097 bash[24958]: cluster 2023-11-17T07:44:54.971653+0000 mgr.smithi097.swvonl (mgr.14180) 2525 : cluster [DBG] pgmap v1776: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:56.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:56 smithi097 bash[24958]: audit 2023-11-17T07:44:55.476183+0000 mon.smithi097 (mon.0) 1318 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:56.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:56 smithi193 bash[28559]: cluster 2023-11-17T07:44:54.971653+0000 mgr.smithi097.swvonl (mgr.14180) 2525 : cluster [DBG] pgmap v1776: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-11-17T07:44:56.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:56 smithi193 bash[28559]: audit 2023-11-17T07:44:55.476183+0000 mon.smithi097 (mon.0) 1318 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:44:56.857 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:44:56.858 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:44:57.785 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:44:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:58 smithi097 bash[24958]: audit 2023-11-17T07:44:56.845886+0000 mgr.smithi097.swvonl (mgr.14180) 2526 : audit [DBG] from='client.15688 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:44:58.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:44:58 smithi097 bash[24958]: cluster 2023-11-17T07:44:56.972628+0000 mgr.smithi097.swvonl (mgr.14180) 2527 : cluster [DBG] pgmap v1777: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:58.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:58 smithi193 bash[28559]: audit 2023-11-17T07:44:56.845886+0000 mgr.smithi097.swvonl (mgr.14180) 2526 : audit [DBG] from='client.15688 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:44:58.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:44:58 smithi193 bash[28559]: cluster 2023-11-17T07:44:56.972628+0000 mgr.smithi097.swvonl (mgr.14180) 2527 : cluster [DBG] pgmap v1777: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:44:58.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:00.514 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:00 smithi097 bash[24958]: cluster 2023-11-17T07:44:58.973612+0000 mgr.smithi097.swvonl (mgr.14180) 2528 : cluster [DBG] pgmap v1778: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:45:00.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:00 smithi193 bash[28559]: cluster 2023-11-17T07:44:58.973612+0000 mgr.smithi097.swvonl (mgr.14180) 2528 : cluster [DBG] pgmap v1778: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:45:01.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:01 smithi097 bash[24958]: cluster 2023-11-17T07:45:00.975038+0000 mgr.smithi097.swvonl (mgr.14180) 2529 : cluster [DBG] pgmap v1779: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:45:01.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:01 smithi193 bash[28559]: cluster 2023-11-17T07:45:00.975038+0000 mgr.smithi097.swvonl (mgr.14180) 2529 : cluster [DBG] pgmap v1779: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 348 B/s rd, 0 op/s 2023-11-17T07:45:02.779 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:02.780 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:03.661 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:04.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:04 smithi097 bash[24958]: audit 2023-11-17T07:45:02.768861+0000 mgr.smithi097.swvonl (mgr.14180) 2530 : audit [DBG] from='client.15692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:04.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:04 smithi097 bash[24958]: cluster 2023-11-17T07:45:02.976079+0000 mgr.smithi097.swvonl (mgr.14180) 2531 : cluster [DBG] pgmap v1780: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:45:04.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:04 smithi193 bash[28559]: audit 2023-11-17T07:45:02.768861+0000 mgr.smithi097.swvonl (mgr.14180) 2530 : audit [DBG] from='client.15692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:04.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:04 smithi193 bash[28559]: cluster 2023-11-17T07:45:02.976079+0000 mgr.smithi097.swvonl (mgr.14180) 2531 : cluster [DBG] pgmap v1780: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2023-11-17T07:45:04.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:06.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:06 smithi097 bash[24958]: cluster 2023-11-17T07:45:04.977530+0000 mgr.smithi097.swvonl (mgr.14180) 2532 : cluster [DBG] pgmap v1781: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:06.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:06 smithi193 bash[28559]: cluster 2023-11-17T07:45:04.977530+0000 mgr.smithi097.swvonl (mgr.14180) 2532 : cluster [DBG] pgmap v1781: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:08.271 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:08.272 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:08.536 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:08 smithi097 bash[24958]: cluster 2023-11-17T07:45:06.978835+0000 mgr.smithi097.swvonl (mgr.14180) 2533 : cluster [DBG] pgmap v1782: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:08.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:08 smithi193 bash[28559]: cluster 2023-11-17T07:45:06.978835+0000 mgr.smithi097.swvonl (mgr.14180) 2533 : cluster [DBG] pgmap v1782: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:09.187 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:09.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:09 smithi097 bash[24958]: audit 2023-11-17T07:45:08.260409+0000 mgr.smithi097.swvonl (mgr.14180) 2534 : audit [DBG] from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:09.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:09 smithi193 bash[28559]: audit 2023-11-17T07:45:08.260409+0000 mgr.smithi097.swvonl (mgr.14180) 2534 : audit [DBG] from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:10.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:10 smithi097 bash[24958]: cluster 2023-11-17T07:45:08.979956+0000 mgr.smithi097.swvonl (mgr.14180) 2535 : cluster [DBG] pgmap v1783: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:10 smithi193 bash[28559]: cluster 2023-11-17T07:45:08.979956+0000 mgr.smithi097.swvonl (mgr.14180) 2535 : cluster [DBG] pgmap v1783: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:12.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:12 smithi097 bash[24958]: cluster 2023-11-17T07:45:10.980825+0000 mgr.smithi097.swvonl (mgr.14180) 2536 : cluster [DBG] pgmap v1784: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:12.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:12 smithi193 bash[28559]: cluster 2023-11-17T07:45:10.980825+0000 mgr.smithi097.swvonl (mgr.14180) 2536 : cluster [DBG] pgmap v1784: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:13.919 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:13.920 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:14.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:14 smithi097 bash[24958]: cluster 2023-11-17T07:45:12.981734+0000 mgr.smithi097.swvonl (mgr.14180) 2537 : cluster [DBG] pgmap v1785: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:14.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:14 smithi193 bash[28559]: cluster 2023-11-17T07:45:12.981734+0000 mgr.smithi097.swvonl (mgr.14180) 2537 : cluster [DBG] pgmap v1785: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:14.910 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:15.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:15 smithi097 bash[24958]: audit 2023-11-17T07:45:13.909434+0000 mgr.smithi097.swvonl (mgr.14180) 2538 : audit [DBG] from='client.15700 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:15.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:15 smithi193 bash[28559]: audit 2023-11-17T07:45:13.909434+0000 mgr.smithi097.swvonl (mgr.14180) 2538 : audit [DBG] from='client.15700 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:15.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:16.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:16 smithi097 bash[24958]: cluster 2023-11-17T07:45:14.983225+0000 mgr.smithi097.swvonl (mgr.14180) 2539 : cluster [DBG] pgmap v1786: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:16.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:16 smithi193 bash[28559]: cluster 2023-11-17T07:45:14.983225+0000 mgr.smithi097.swvonl (mgr.14180) 2539 : cluster [DBG] pgmap v1786: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:18.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:18 smithi097 bash[24958]: cluster 2023-11-17T07:45:16.984573+0000 mgr.smithi097.swvonl (mgr.14180) 2540 : cluster [DBG] pgmap v1787: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:18.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:18 smithi193 bash[28559]: cluster 2023-11-17T07:45:16.984573+0000 mgr.smithi097.swvonl (mgr.14180) 2540 : cluster [DBG] pgmap v1787: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:19.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:19.954 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:20.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:20 smithi097 bash[24958]: cluster 2023-11-17T07:45:18.985731+0000 mgr.smithi097.swvonl (mgr.14180) 2541 : cluster [DBG] pgmap v1788: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:20.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:20 smithi193 bash[28559]: cluster 2023-11-17T07:45:18.985731+0000 mgr.smithi097.swvonl (mgr.14180) 2541 : cluster [DBG] pgmap v1788: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:20.927 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:21.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:21 smithi097 bash[24958]: audit 2023-11-17T07:45:19.942014+0000 mgr.smithi097.swvonl (mgr.14180) 2542 : audit [DBG] from='client.15704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:21 smithi193 bash[28559]: audit 2023-11-17T07:45:19.942014+0000 mgr.smithi097.swvonl (mgr.14180) 2542 : audit [DBG] from='client.15704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:21.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:22.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:22 smithi097 bash[24958]: cluster 2023-11-17T07:45:20.987116+0000 mgr.smithi097.swvonl (mgr.14180) 2543 : cluster [DBG] pgmap v1789: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:22.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:22 smithi193 bash[28559]: cluster 2023-11-17T07:45:20.987116+0000 mgr.smithi097.swvonl (mgr.14180) 2543 : cluster [DBG] pgmap v1789: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:24.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:24 smithi097 bash[24958]: cluster 2023-11-17T07:45:22.988035+0000 mgr.smithi097.swvonl (mgr.14180) 2544 : cluster [DBG] pgmap v1790: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:24.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:24 smithi193 bash[28559]: cluster 2023-11-17T07:45:22.988035+0000 mgr.smithi097.swvonl (mgr.14180) 2544 : cluster [DBG] pgmap v1790: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:25.982 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:25.983 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:26.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:26 smithi097 bash[24958]: cluster 2023-11-17T07:45:24.989148+0000 mgr.smithi097.swvonl (mgr.14180) 2545 : cluster [DBG] pgmap v1791: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:26.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:26 smithi193 bash[28559]: cluster 2023-11-17T07:45:24.989148+0000 mgr.smithi097.swvonl (mgr.14180) 2545 : cluster [DBG] pgmap v1791: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:26.944 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:27.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:27 smithi097 bash[24958]: audit 2023-11-17T07:45:25.971136+0000 mgr.smithi097.swvonl (mgr.14180) 2546 : audit [DBG] from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:27.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:27 smithi097 bash[24958]: cluster 2023-11-17T07:45:26.990202+0000 mgr.smithi097.swvonl (mgr.14180) 2547 : cluster [DBG] pgmap v1792: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:27 smithi193 bash[28559]: audit 2023-11-17T07:45:25.971136+0000 mgr.smithi097.swvonl (mgr.14180) 2546 : audit [DBG] from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:27 smithi193 bash[28559]: cluster 2023-11-17T07:45:26.990202+0000 mgr.smithi097.swvonl (mgr.14180) 2547 : cluster [DBG] pgmap v1792: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:27.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:30.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:30 smithi193 bash[28559]: cluster 2023-11-17T07:45:28.991299+0000 mgr.smithi097.swvonl (mgr.14180) 2548 : cluster [DBG] pgmap v1793: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:30.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:30 smithi097 bash[24958]: cluster 2023-11-17T07:45:28.991299+0000 mgr.smithi097.swvonl (mgr.14180) 2548 : cluster [DBG] pgmap v1793: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:31.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:31.954 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:32.267 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:32 smithi097 bash[24958]: cluster 2023-11-17T07:45:30.992727+0000 mgr.smithi097.swvonl (mgr.14180) 2549 : cluster [DBG] pgmap v1794: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:32.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:32 smithi193 bash[28559]: cluster 2023-11-17T07:45:30.992727+0000 mgr.smithi097.swvonl (mgr.14180) 2549 : cluster [DBG] pgmap v1794: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:32.966 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:33.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:33 smithi097 bash[24958]: audit 2023-11-17T07:45:31.941894+0000 mgr.smithi097.swvonl (mgr.14180) 2550 : audit [DBG] from='client.15712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:33.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:33 smithi193 bash[28559]: audit 2023-11-17T07:45:31.941894+0000 mgr.smithi097.swvonl (mgr.14180) 2550 : audit [DBG] from='client.15712 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:33.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:34.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:34 smithi097 bash[24958]: cluster 2023-11-17T07:45:32.993565+0000 mgr.smithi097.swvonl (mgr.14180) 2551 : cluster [DBG] pgmap v1795: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:34.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:34 smithi193 bash[28559]: cluster 2023-11-17T07:45:32.993565+0000 mgr.smithi097.swvonl (mgr.14180) 2551 : cluster [DBG] pgmap v1795: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:36.530 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:36 smithi097 bash[24958]: cluster 2023-11-17T07:45:34.994973+0000 mgr.smithi097.swvonl (mgr.14180) 2552 : cluster [DBG] pgmap v1796: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.287 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:45:55.314 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:43:44.912679Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.serrsg on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:45:55.319 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:38 smithi097 bash[24958]: cluster 2023-11-17T07:45:36.996045+0000 mgr.smithi097.swvonl (mgr.14180) 2553 : cluster [DBG] pgmap v1797: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.321 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:45:55.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:36 smithi193 bash[28559]: cluster 2023-11-17T07:45:34.994973+0000 mgr.smithi097.swvonl (mgr.14180) 2552 : cluster [DBG] pgmap v1796: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.321 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:38 smithi193 bash[28559]: cluster 2023-11-17T07:45:36.996045+0000 mgr.smithi097.swvonl (mgr.14180) 2553 : cluster [DBG] pgmap v1797: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:39 smithi193 bash[28559]: audit 2023-11-17T07:45:37.980740+0000 mgr.smithi097.swvonl (mgr.14180) 2554 : audit [DBG] from='client.15716 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:40 smithi193 bash[28559]: cluster 2023-11-17T07:45:38.997183+0000 mgr.smithi097.swvonl (mgr.14180) 2555 : cluster [DBG] pgmap v1798: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:42 smithi193 bash[28559]: cluster 2023-11-17T07:45:40.998133+0000 mgr.smithi097.swvonl (mgr.14180) 2556 : cluster [DBG] pgmap v1799: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:44 smithi193 bash[28559]: cluster 2023-11-17T07:45:42.999069+0000 mgr.smithi097.swvonl (mgr.14180) 2557 : cluster [DBG] pgmap v1800: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:46 smithi193 bash[28559]: cluster 2023-11-17T07:45:45.000471+0000 mgr.smithi097.swvonl (mgr.14180) 2558 : cluster [DBG] pgmap v1801: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:48 smithi193 bash[28559]: cluster 2023-11-17T07:45:47.001590+0000 mgr.smithi097.swvonl (mgr.14180) 2559 : cluster [DBG] pgmap v1802: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:50 smithi193 bash[28559]: cluster 2023-11-17T07:45:49.002512+0000 mgr.smithi097.swvonl (mgr.14180) 2560 : cluster [DBG] pgmap v1803: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:52 smithi193 bash[28559]: cluster 2023-11-17T07:45:51.003892+0000 mgr.smithi097.swvonl (mgr.14180) 2561 : cluster [DBG] pgmap v1804: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:53 smithi193 bash[28559]: audit 2023-11-17T07:45:52.974690+0000 mon.smithi097 (mon.0) 1319 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:53 smithi193 bash[28559]: audit 2023-11-17T07:45:53.281028+0000 mon.smithi097 (mon.0) 1320 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:54 smithi193 bash[28559]: cluster 2023-11-17T07:45:53.004872+0000 mgr.smithi097.swvonl (mgr.14180) 2562 : cluster [DBG] pgmap v1805: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.322 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:54 smithi193 bash[28559]: audit 2023-11-17T07:45:53.282484+0000 mon.smithi097 (mon.0) 1321 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:45:55.323 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:54 smithi193 bash[28559]: audit 2023-11-17T07:45:53.296181+0000 mon.smithi097 (mon.0) 1322 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:55.323 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:54 smithi193 bash[28559]: audit 2023-11-17T07:45:53.303728+0000 mon.smithi097 (mon.0) 1323 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:45:55.323 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:54 smithi193 bash[28559]: audit 2023-11-17T07:45:53.318476+0000 mon.smithi097 (mon.0) 1324 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:55.323 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:39 smithi097 bash[24958]: audit 2023-11-17T07:45:37.980740+0000 mgr.smithi097.swvonl (mgr.14180) 2554 : audit [DBG] from='client.15716 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:40 smithi097 bash[24958]: cluster 2023-11-17T07:45:38.997183+0000 mgr.smithi097.swvonl (mgr.14180) 2555 : cluster [DBG] pgmap v1798: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:42 smithi097 bash[24958]: cluster 2023-11-17T07:45:40.998133+0000 mgr.smithi097.swvonl (mgr.14180) 2556 : cluster [DBG] pgmap v1799: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:44 smithi097 bash[24958]: cluster 2023-11-17T07:45:42.999069+0000 mgr.smithi097.swvonl (mgr.14180) 2557 : cluster [DBG] pgmap v1800: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:46 smithi097 bash[24958]: cluster 2023-11-17T07:45:45.000471+0000 mgr.smithi097.swvonl (mgr.14180) 2558 : cluster [DBG] pgmap v1801: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:48 smithi097 bash[24958]: cluster 2023-11-17T07:45:47.001590+0000 mgr.smithi097.swvonl (mgr.14180) 2559 : cluster [DBG] pgmap v1802: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:50 smithi097 bash[24958]: cluster 2023-11-17T07:45:49.002512+0000 mgr.smithi097.swvonl (mgr.14180) 2560 : cluster [DBG] pgmap v1803: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:52 smithi097 bash[24958]: cluster 2023-11-17T07:45:51.003892+0000 mgr.smithi097.swvonl (mgr.14180) 2561 : cluster [DBG] pgmap v1804: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:53 smithi097 bash[24958]: audit 2023-11-17T07:45:52.974690+0000 mon.smithi097 (mon.0) 1319 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:53 smithi097 bash[24958]: audit 2023-11-17T07:45:53.281028+0000 mon.smithi097 (mon.0) 1320 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:54 smithi097 bash[24958]: cluster 2023-11-17T07:45:53.004872+0000 mgr.smithi097.swvonl (mgr.14180) 2562 : cluster [DBG] pgmap v1805: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:54 smithi097 bash[24958]: audit 2023-11-17T07:45:53.282484+0000 mon.smithi097 (mon.0) 1321 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:54 smithi097 bash[24958]: audit 2023-11-17T07:45:53.296181+0000 mon.smithi097 (mon.0) 1322 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:55.324 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:54 smithi097 bash[24958]: audit 2023-11-17T07:45:53.303728+0000 mon.smithi097 (mon.0) 1323 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:45:55.325 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:54 smithi097 bash[24958]: audit 2023-11-17T07:45:53.318476+0000 mon.smithi097 (mon.0) 1324 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:55.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:55 smithi193 bash[28559]: cluster 2023-11-17T07:45:53.297012+0000 mgr.smithi097.swvonl (mgr.14180) 2563 : cluster [DBG] pgmap v1806: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:55 smithi193 bash[28559]: cephadm 2023-11-17T07:45:53.324421+0000 mgr.smithi097.swvonl (mgr.14180) 2564 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ezysox on smithi193 2023-11-17T07:45:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:55 smithi193 bash[28559]: cluster 2023-11-17T07:45:54.299087+0000 mon.smithi097 (mon.0) 1325 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:45:55.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:55 smithi193 bash[28559]: cluster 2023-11-17T07:45:54.299134+0000 mon.smithi097 (mon.0) 1326 : cluster [INF] Cluster is now healthy 2023-11-17T07:45:55.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:55 smithi097 bash[24958]: cluster 2023-11-17T07:45:53.297012+0000 mgr.smithi097.swvonl (mgr.14180) 2563 : cluster [DBG] pgmap v1806: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:55.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:55 smithi097 bash[24958]: cephadm 2023-11-17T07:45:53.324421+0000 mgr.smithi097.swvonl (mgr.14180) 2564 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.ezysox on smithi193 2023-11-17T07:45:55.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:55 smithi097 bash[24958]: cluster 2023-11-17T07:45:54.299087+0000 mon.smithi097 (mon.0) 1325 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:45:55.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:55 smithi097 bash[24958]: cluster 2023-11-17T07:45:54.299134+0000 mon.smithi097 (mon.0) 1326 : cluster [INF] Cluster is now healthy 2023-11-17T07:45:56.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:45:56.771 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:56 smithi193 bash[28559]: cluster 2023-11-17T07:45:55.298330+0000 mgr.smithi097.swvonl (mgr.14180) 2565 : cluster [DBG] pgmap v1807: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:56.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:56 smithi097 bash[24958]: cluster 2023-11-17T07:45:55.298330+0000 mgr.smithi097.swvonl (mgr.14180) 2565 : cluster [DBG] pgmap v1807: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:57.699 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:57 smithi097 bash[24958]: audit 2023-11-17T07:45:56.587790+0000 mon.smithi097 (mon.0) 1327 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:57.700 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:57 smithi097 bash[24958]: audit 2023-11-17T07:45:57.437251+0000 mon.smithi097 (mon.0) 1328 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ezysox"}]: dispatch 2023-11-17T07:45:57.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:57 smithi193 bash[28559]: audit 2023-11-17T07:45:56.587790+0000 mon.smithi097 (mon.0) 1327 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:45:57.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:57 smithi193 bash[28559]: audit 2023-11-17T07:45:57.437251+0000 mon.smithi097 (mon.0) 1328 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.ezysox"}]: dispatch 2023-11-17T07:45:59.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: cluster 2023-11-17T07:45:57.298836+0000 mgr.smithi097.swvonl (mgr.14180) 2566 : cluster [DBG] pgmap v1808: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:59.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: cephadm 2023-11-17T07:45:57.436252+0000 mgr.smithi097.swvonl (mgr.14180) 2567 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: cephadm 2023-11-17T07:45:57.436669+0000 mgr.smithi097.swvonl (mgr.14180) 2568 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: cephadm 2023-11-17T07:45:57.438705+0000 mgr.smithi097.swvonl (mgr.14180) 2569 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:45:59 smithi097 bash[24958]: cephadm 2023-11-17T07:45:57.442699+0000 mgr.smithi097.swvonl (mgr.14180) 2570 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.qsjxxv on smithi097 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: cluster 2023-11-17T07:45:57.298836+0000 mgr.smithi097.swvonl (mgr.14180) 2566 : cluster [DBG] pgmap v1808: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: cephadm 2023-11-17T07:45:57.436252+0000 mgr.smithi097.swvonl (mgr.14180) 2567 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: cephadm 2023-11-17T07:45:57.436669+0000 mgr.smithi097.swvonl (mgr.14180) 2568 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: cephadm 2023-11-17T07:45:57.438705+0000 mgr.smithi097.swvonl (mgr.14180) 2569 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.ezysox ... 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:45:59.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:45:59 smithi193 bash[28559]: cephadm 2023-11-17T07:45:57.442699+0000 mgr.smithi097.swvonl (mgr.14180) 2570 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.qsjxxv on smithi097 2023-11-17T07:46:00.161 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:00.161 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:49.220044Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.iyjktt on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-iyjktt\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.iyjktt\nDeploy daemon haproxy.nfs.foo.smithi193.iyjktt ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.963340Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.fninro on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-fninro\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.fninro\nDeploy daemon haproxy.nfs.foo.smithi097.fninro ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:44:52.966629Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.fyncxx on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:01.235 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:01.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:01 smithi097 bash[24958]: cluster 2023-11-17T07:45:59.299668+0000 mgr.smithi097.swvonl (mgr.14180) 2571 : cluster [DBG] pgmap v1809: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:46:01.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:01 smithi097 bash[24958]: audit 2023-11-17T07:46:00.147514+0000 mgr.smithi097.swvonl (mgr.14180) 2572 : audit [DBG] from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:01.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:01 smithi193 bash[28559]: cluster 2023-11-17T07:45:59.299668+0000 mgr.smithi097.swvonl (mgr.14180) 2571 : cluster [DBG] pgmap v1809: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:46:01.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:01 smithi193 bash[28559]: audit 2023-11-17T07:46:00.147514+0000 mgr.smithi097.swvonl (mgr.14180) 2572 : audit [DBG] from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:02.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:02.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:02 smithi097 bash[24958]: audit 2023-11-17T07:46:01.336993+0000 mon.smithi097 (mon.0) 1329 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.qsjxxv"}]: dispatch 2023-11-17T07:46:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:02 smithi193 bash[28559]: audit 2023-11-17T07:46:01.336993+0000 mon.smithi097 (mon.0) 1329 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.qsjxxv"}]: dispatch 2023-11-17T07:46:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cluster 2023-11-17T07:46:01.300572+0000 mgr.smithi097.swvonl (mgr.14180) 2573 : cluster [DBG] pgmap v1810: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:46:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cephadm 2023-11-17T07:46:01.335710+0000 mgr.smithi097.swvonl (mgr.14180) 2574 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.550 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.551 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cephadm 2023-11-17T07:46:01.336258+0000 mgr.smithi097.swvonl (mgr.14180) 2575 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cephadm 2023-11-17T07:46:01.338745+0000 mgr.smithi097.swvonl (mgr.14180) 2576 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.552 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cephadm 2023-11-17T07:46:01.343084+0000 mgr.smithi097.swvonl (mgr.14180) 2577 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cephadm 2023-11-17T07:46:01.344701+0000 mgr.smithi097.swvonl (mgr.14180) 2578 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cluster 2023-11-17T07:46:01.345574+0000 mgr.smithi097.swvonl (mgr.14180) 2579 : cluster [DBG] pgmap v1811: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 245 B/s rd, 0 op/s 2023-11-17T07:46:03.553 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:03 smithi097 bash[24958]: cluster 2023-11-17T07:46:02.234852+0000 mon.smithi097 (mon.0) 1330 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cluster 2023-11-17T07:46:01.300572+0000 mgr.smithi097.swvonl (mgr.14180) 2573 : cluster [DBG] pgmap v1810: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cephadm 2023-11-17T07:46:01.335710+0000 mgr.smithi097.swvonl (mgr.14180) 2574 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.557 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cephadm 2023-11-17T07:46:01.336258+0000 mgr.smithi097.swvonl (mgr.14180) 2575 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cephadm 2023-11-17T07:46:01.338745+0000 mgr.smithi097.swvonl (mgr.14180) 2576 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.qsjxxv ... 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:46:03.559 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cephadm 2023-11-17T07:46:01.343084+0000 mgr.smithi097.swvonl (mgr.14180) 2577 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cephadm 2023-11-17T07:46:01.344701+0000 mgr.smithi097.swvonl (mgr.14180) 2578 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cluster 2023-11-17T07:46:01.345574+0000 mgr.smithi097.swvonl (mgr.14180) 2579 : cluster [DBG] pgmap v1811: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 245 B/s rd, 0 op/s 2023-11-17T07:46:03.560 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:03 smithi193 bash[28559]: cluster 2023-11-17T07:46:02.234852+0000 mon.smithi097 (mon.0) 1330 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:46:05.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:05 smithi097 bash[24958]: cluster 2023-11-17T07:46:03.346675+0000 mgr.smithi097.swvonl (mgr.14180) 2580 : cluster [DBG] pgmap v1812: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:05.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:05 smithi193 bash[28559]: cluster 2023-11-17T07:46:03.346675+0000 mgr.smithi097.swvonl (mgr.14180) 2580 : cluster [DBG] pgmap v1812: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:06.325 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:06.325 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:06.612 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:06 smithi097 bash[24958]: cluster 2023-11-17T07:46:05.348087+0000 mgr.smithi097.swvonl (mgr.14180) 2581 : cluster [DBG] pgmap v1813: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:06.612 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:06 smithi097 bash[24958]: audit 2023-11-17T07:46:05.529809+0000 mon.smithi097 (mon.0) 1331 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:46:06.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:06 smithi193 bash[28559]: cluster 2023-11-17T07:46:05.348087+0000 mgr.smithi097.swvonl (mgr.14180) 2581 : cluster [DBG] pgmap v1813: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:06.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:06 smithi193 bash[28559]: audit 2023-11-17T07:46:05.529809+0000 mon.smithi097 (mon.0) 1331 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:46:07.181 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:08.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:08.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:08 smithi097 bash[24958]: audit 2023-11-17T07:46:06.312885+0000 mgr.smithi097.swvonl (mgr.14180) 2582 : audit [DBG] from='client.15724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:08.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:08 smithi193 bash[28559]: audit 2023-11-17T07:46:06.312885+0000 mgr.smithi097.swvonl (mgr.14180) 2582 : audit [DBG] from='client.15724 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:09.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:09 smithi097 bash[24958]: cluster 2023-11-17T07:46:07.349071+0000 mgr.smithi097.swvonl (mgr.14180) 2583 : cluster [DBG] pgmap v1814: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:09.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:09 smithi193 bash[28559]: cluster 2023-11-17T07:46:07.349071+0000 mgr.smithi097.swvonl (mgr.14180) 2583 : cluster [DBG] pgmap v1814: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:11.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:11 smithi097 bash[24958]: cluster 2023-11-17T07:46:09.350155+0000 mgr.smithi097.swvonl (mgr.14180) 2584 : cluster [DBG] pgmap v1815: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:11.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:11 smithi193 bash[28559]: cluster 2023-11-17T07:46:09.350155+0000 mgr.smithi097.swvonl (mgr.14180) 2584 : cluster [DBG] pgmap v1815: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:12.035 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:12.035 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:12 smithi097 bash[24958]: cluster 2023-11-17T07:46:11.351469+0000 mgr.smithi097.swvonl (mgr.14180) 2585 : cluster [DBG] pgmap v1816: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:12 smithi097 bash[24958]: audit 2023-11-17T07:46:11.588299+0000 mon.smithi097 (mon.0) 1332 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:46:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:12 smithi097 bash[24958]: audit 2023-11-17T07:46:12.023776+0000 mgr.smithi097.swvonl (mgr.14180) 2586 : audit [DBG] from='client.15728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:13.020 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:12 smithi193 bash[28559]: cluster 2023-11-17T07:46:11.351469+0000 mgr.smithi097.swvonl (mgr.14180) 2585 : cluster [DBG] pgmap v1816: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 203 B/s rd, 0 op/s 2023-11-17T07:46:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:12 smithi193 bash[28559]: audit 2023-11-17T07:46:11.588299+0000 mon.smithi097 (mon.0) 1332 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:46:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:12 smithi193 bash[28559]: audit 2023-11-17T07:46:12.023776+0000 mgr.smithi097.swvonl (mgr.14180) 2586 : audit [DBG] from='client.15728 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:14.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:15.048 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:14 smithi097 bash[24958]: cluster 2023-11-17T07:46:13.352548+0000 mgr.smithi097.swvonl (mgr.14180) 2587 : cluster [DBG] pgmap v1817: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:15.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:14 smithi193 bash[28559]: cluster 2023-11-17T07:46:13.352548+0000 mgr.smithi097.swvonl (mgr.14180) 2587 : cluster [DBG] pgmap v1817: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:16.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:16 smithi097 bash[24958]: cluster 2023-11-17T07:46:15.354007+0000 mgr.smithi097.swvonl (mgr.14180) 2588 : cluster [DBG] pgmap v1818: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:17.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:16 smithi193 bash[28559]: cluster 2023-11-17T07:46:15.354007+0000 mgr.smithi097.swvonl (mgr.14180) 2588 : cluster [DBG] pgmap v1818: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:18.012 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:18.012 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:18.938 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:19.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:19 smithi097 bash[24958]: cluster 2023-11-17T07:46:17.355052+0000 mgr.smithi097.swvonl (mgr.14180) 2589 : cluster [DBG] pgmap v1819: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:19.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:19 smithi097 bash[24958]: audit 2023-11-17T07:46:18.005029+0000 mgr.smithi097.swvonl (mgr.14180) 2590 : audit [DBG] from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:19 smithi193 bash[28559]: cluster 2023-11-17T07:46:17.355052+0000 mgr.smithi097.swvonl (mgr.14180) 2589 : cluster [DBG] pgmap v1819: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:19 smithi193 bash[28559]: audit 2023-11-17T07:46:18.005029+0000 mgr.smithi097.swvonl (mgr.14180) 2590 : audit [DBG] from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:19.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:21.529 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:21 smithi097 bash[24958]: cluster 2023-11-17T07:46:19.356161+0000 mgr.smithi097.swvonl (mgr.14180) 2591 : cluster [DBG] pgmap v1820: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:21 smithi193 bash[28559]: cluster 2023-11-17T07:46:19.356161+0000 mgr.smithi097.swvonl (mgr.14180) 2591 : cluster [DBG] pgmap v1820: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:23.527 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:23 smithi097 bash[24958]: cluster 2023-11-17T07:46:21.357527+0000 mgr.smithi097.swvonl (mgr.14180) 2592 : cluster [DBG] pgmap v1821: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:23.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:23 smithi193 bash[28559]: cluster 2023-11-17T07:46:21.357527+0000 mgr.smithi097.swvonl (mgr.14180) 2592 : cluster [DBG] pgmap v1821: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:23.897 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:23.898 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:24.853 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:25.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:25 smithi097 bash[24958]: cluster 2023-11-17T07:46:23.358276+0000 mgr.smithi097.swvonl (mgr.14180) 2593 : cluster [DBG] pgmap v1822: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:25.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:25 smithi097 bash[24958]: audit 2023-11-17T07:46:23.889606+0000 mgr.smithi097.swvonl (mgr.14180) 2594 : audit [DBG] from='client.15736 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:25 smithi193 bash[28559]: cluster 2023-11-17T07:46:23.358276+0000 mgr.smithi097.swvonl (mgr.14180) 2593 : cluster [DBG] pgmap v1822: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:25 smithi193 bash[28559]: audit 2023-11-17T07:46:23.889606+0000 mgr.smithi097.swvonl (mgr.14180) 2594 : audit [DBG] from='client.15736 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:25.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:27.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:27 smithi097 bash[24958]: cluster 2023-11-17T07:46:25.359690+0000 mgr.smithi097.swvonl (mgr.14180) 2595 : cluster [DBG] pgmap v1823: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:27 smithi193 bash[28559]: cluster 2023-11-17T07:46:25.359690+0000 mgr.smithi097.swvonl (mgr.14180) 2595 : cluster [DBG] pgmap v1823: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:29.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:29 smithi097 bash[24958]: cluster 2023-11-17T07:46:27.360581+0000 mgr.smithi097.swvonl (mgr.14180) 2596 : cluster [DBG] pgmap v1824: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:29.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:29 smithi193 bash[28559]: cluster 2023-11-17T07:46:27.360581+0000 mgr.smithi097.swvonl (mgr.14180) 2596 : cluster [DBG] pgmap v1824: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:29.697 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:29.698 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:30.620 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:31.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:31 smithi097 bash[24958]: cluster 2023-11-17T07:46:29.361663+0000 mgr.smithi097.swvonl (mgr.14180) 2597 : cluster [DBG] pgmap v1825: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:31.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:31 smithi097 bash[24958]: audit 2023-11-17T07:46:29.690673+0000 mgr.smithi097.swvonl (mgr.14180) 2598 : audit [DBG] from='client.15740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:31.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:31 smithi193 bash[28559]: cluster 2023-11-17T07:46:29.361663+0000 mgr.smithi097.swvonl (mgr.14180) 2597 : cluster [DBG] pgmap v1825: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:31.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:31 smithi193 bash[28559]: audit 2023-11-17T07:46:29.690673+0000 mgr.smithi097.swvonl (mgr.14180) 2598 : audit [DBG] from='client.15740 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:31.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:33.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:33 smithi097 bash[24958]: cluster 2023-11-17T07:46:31.363083+0000 mgr.smithi097.swvonl (mgr.14180) 2599 : cluster [DBG] pgmap v1826: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:33.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:33 smithi193 bash[28559]: cluster 2023-11-17T07:46:31.363083+0000 mgr.smithi097.swvonl (mgr.14180) 2599 : cluster [DBG] pgmap v1826: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:35.407 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:35.407 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:35.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:35 smithi097 bash[24958]: cluster 2023-11-17T07:46:33.364122+0000 mgr.smithi097.swvonl (mgr.14180) 2600 : cluster [DBG] pgmap v1827: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:35.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:35 smithi193 bash[28559]: cluster 2023-11-17T07:46:33.364122+0000 mgr.smithi097.swvonl (mgr.14180) 2600 : cluster [DBG] pgmap v1827: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:36.352 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:37.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:37.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:37 smithi097 bash[24958]: cluster 2023-11-17T07:46:35.365416+0000 mgr.smithi097.swvonl (mgr.14180) 2601 : cluster [DBG] pgmap v1828: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:37.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:37 smithi097 bash[24958]: audit 2023-11-17T07:46:35.396049+0000 mgr.smithi097.swvonl (mgr.14180) 2602 : audit [DBG] from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:37.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:37 smithi193 bash[28559]: cluster 2023-11-17T07:46:35.365416+0000 mgr.smithi097.swvonl (mgr.14180) 2601 : cluster [DBG] pgmap v1828: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:37.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:37 smithi193 bash[28559]: audit 2023-11-17T07:46:35.396049+0000 mgr.smithi097.swvonl (mgr.14180) 2602 : audit [DBG] from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:39.298 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:39 smithi097 bash[24958]: cluster 2023-11-17T07:46:37.366503+0000 mgr.smithi097.swvonl (mgr.14180) 2603 : cluster [DBG] pgmap v1829: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:39.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:39 smithi193 bash[28559]: cluster 2023-11-17T07:46:37.366503+0000 mgr.smithi097.swvonl (mgr.14180) 2603 : cluster [DBG] pgmap v1829: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:41.287 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:41.287 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:41.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:41 smithi097 bash[24958]: cluster 2023-11-17T07:46:39.367697+0000 mgr.smithi097.swvonl (mgr.14180) 2604 : cluster [DBG] pgmap v1830: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:41.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:41 smithi193 bash[28559]: cluster 2023-11-17T07:46:39.367697+0000 mgr.smithi097.swvonl (mgr.14180) 2604 : cluster [DBG] pgmap v1830: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:42.306 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:42 smithi193 bash[28559]: audit 2023-11-17T07:46:41.275770+0000 mgr.smithi097.swvonl (mgr.14180) 2605 : audit [DBG] from='client.15748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:43 smithi193 bash[28559]: cluster 2023-11-17T07:46:41.369283+0000 mgr.smithi097.swvonl (mgr.14180) 2606 : cluster [DBG] pgmap v1831: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:45 smithi193 bash[28559]: cluster 2023-11-17T07:46:43.370372+0000 mgr.smithi097.swvonl (mgr.14180) 2607 : cluster [DBG] pgmap v1832: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:46 smithi193 bash[28559]: cluster 2023-11-17T07:46:45.371847+0000 mgr.smithi097.swvonl (mgr.14180) 2608 : cluster [DBG] pgmap v1833: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:49 smithi193 bash[28559]: cluster 2023-11-17T07:46:47.372899+0000 mgr.smithi097.swvonl (mgr.14180) 2609 : cluster [DBG] pgmap v1834: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.032 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:51 smithi193 bash[28559]: cluster 2023-11-17T07:46:49.374175+0000 mgr.smithi097.swvonl (mgr.14180) 2610 : cluster [DBG] pgmap v1835: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:42 smithi097 bash[24958]: audit 2023-11-17T07:46:41.275770+0000 mgr.smithi097.swvonl (mgr.14180) 2605 : audit [DBG] from='client.15748 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:43 smithi097 bash[24958]: cluster 2023-11-17T07:46:41.369283+0000 mgr.smithi097.swvonl (mgr.14180) 2606 : cluster [DBG] pgmap v1831: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:45 smithi097 bash[24958]: cluster 2023-11-17T07:46:43.370372+0000 mgr.smithi097.swvonl (mgr.14180) 2607 : cluster [DBG] pgmap v1832: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:46 smithi097 bash[24958]: cluster 2023-11-17T07:46:45.371847+0000 mgr.smithi097.swvonl (mgr.14180) 2608 : cluster [DBG] pgmap v1833: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:49 smithi097 bash[24958]: cluster 2023-11-17T07:46:47.372899+0000 mgr.smithi097.swvonl (mgr.14180) 2609 : cluster [DBG] pgmap v1834: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.033 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:51 smithi097 bash[24958]: cluster 2023-11-17T07:46:49.374175+0000 mgr.smithi097.swvonl (mgr.14180) 2610 : cluster [DBG] pgmap v1835: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:53.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:53 smithi097 bash[24958]: cluster 2023-11-17T07:46:51.375580+0000 mgr.smithi097.swvonl (mgr.14180) 2611 : cluster [DBG] pgmap v1836: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:53.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:53 smithi193 bash[28559]: cluster 2023-11-17T07:46:51.375580+0000 mgr.smithi097.swvonl (mgr.14180) 2611 : cluster [DBG] pgmap v1836: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:46:54.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:46:55.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:55 smithi097 bash[24958]: cluster 2023-11-17T07:46:53.376767+0000 mgr.smithi097.swvonl (mgr.14180) 2612 : cluster [DBG] pgmap v1837: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:55.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:55 smithi193 bash[28559]: cluster 2023-11-17T07:46:53.376767+0000 mgr.smithi097.swvonl (mgr.14180) 2612 : cluster [DBG] pgmap v1837: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:57.543 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:57 smithi097 bash[24958]: cluster 2023-11-17T07:46:55.378163+0000 mgr.smithi097.swvonl (mgr.14180) 2613 : cluster [DBG] pgmap v1838: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:57.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:57 smithi193 bash[28559]: cluster 2023-11-17T07:46:55.378163+0000 mgr.smithi097.swvonl (mgr.14180) 2613 : cluster [DBG] pgmap v1838: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:57.963 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:46:57.963 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:46:58.857 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:46:59.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:59 smithi097 bash[24958]: cluster 2023-11-17T07:46:57.379137+0000 mgr.smithi097.swvonl (mgr.14180) 2614 : cluster [DBG] pgmap v1839: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:59.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:46:59 smithi097 bash[24958]: audit 2023-11-17T07:46:57.952065+0000 mgr.smithi097.swvonl (mgr.14180) 2615 : audit [DBG] from='client.15752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:59.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:59 smithi193 bash[28559]: cluster 2023-11-17T07:46:57.379137+0000 mgr.smithi097.swvonl (mgr.14180) 2614 : cluster [DBG] pgmap v1839: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:46:59.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:46:59 smithi193 bash[28559]: audit 2023-11-17T07:46:57.952065+0000 mgr.smithi097.swvonl (mgr.14180) 2615 : audit [DBG] from='client.15752 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:46:59.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:47:01.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:01 smithi097 bash[24958]: cluster 2023-11-17T07:46:59.380222+0000 mgr.smithi097.swvonl (mgr.14180) 2616 : cluster [DBG] pgmap v1840: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:01.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:01 smithi193 bash[28559]: cluster 2023-11-17T07:46:59.380222+0000 mgr.smithi097.swvonl (mgr.14180) 2616 : cluster [DBG] pgmap v1840: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.348654+0000 mon.smithi097 (mon.0) 1333 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.660895+0000 mon.smithi097 (mon.0) 1334 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.662089+0000 mon.smithi097 (mon.0) 1335 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.668236+0000 mon.smithi097 (mon.0) 1336 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.674922+0000 mon.smithi097 (mon.0) 1337 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:47:02.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:02 smithi097 bash[24958]: audit 2023-11-17T07:47:01.684256+0000 mon.smithi097 (mon.0) 1338 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.348654+0000 mon.smithi097 (mon.0) 1333 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:47:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.660895+0000 mon.smithi097 (mon.0) 1334 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:47:02.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.662089+0000 mon.smithi097 (mon.0) 1335 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:47:02.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.668236+0000 mon.smithi097 (mon.0) 1336 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:02.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.674922+0000 mon.smithi097 (mon.0) 1337 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:47:02.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:02 smithi193 bash[28559]: audit 2023-11-17T07:47:01.684256+0000 mon.smithi097 (mon.0) 1338 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:03.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:03 smithi097 bash[24958]: cluster 2023-11-17T07:47:01.381284+0000 mgr.smithi097.swvonl (mgr.14180) 2617 : cluster [DBG] pgmap v1841: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:47:03.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:03 smithi097 bash[24958]: cluster 2023-11-17T07:47:01.668825+0000 mgr.smithi097.swvonl (mgr.14180) 2618 : cluster [DBG] pgmap v1842: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:03 smithi097 bash[24958]: cephadm 2023-11-17T07:47:01.686793+0000 mgr.smithi097.swvonl (mgr.14180) 2619 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rjtocq on smithi193 2023-11-17T07:47:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:03 smithi097 bash[24958]: cluster 2023-11-17T07:47:02.278386+0000 mon.smithi097 (mon.0) 1339 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:47:03.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:03 smithi097 bash[24958]: cluster 2023-11-17T07:47:02.278419+0000 mon.smithi097 (mon.0) 1340 : cluster [INF] Cluster is now healthy 2023-11-17T07:47:03.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:03 smithi193 bash[28559]: cluster 2023-11-17T07:47:01.381284+0000 mgr.smithi097.swvonl (mgr.14180) 2617 : cluster [DBG] pgmap v1841: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:47:03.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:03 smithi193 bash[28559]: cluster 2023-11-17T07:47:01.668825+0000 mgr.smithi097.swvonl (mgr.14180) 2618 : cluster [DBG] pgmap v1842: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:03.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:03 smithi193 bash[28559]: cephadm 2023-11-17T07:47:01.686793+0000 mgr.smithi097.swvonl (mgr.14180) 2619 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.rjtocq on smithi193 2023-11-17T07:47:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:03 smithi193 bash[28559]: cluster 2023-11-17T07:47:02.278386+0000 mon.smithi097 (mon.0) 1339 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:47:03.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:03 smithi193 bash[28559]: cluster 2023-11-17T07:47:02.278419+0000 mon.smithi097 (mon.0) 1340 : cluster [INF] Cluster is now healthy 2023-11-17T07:47:03.923 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:47:03.924 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:44:52.969317Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.hrqczz on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:45:57.438473Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.ezysox on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-ezysox\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.ezysox\nDeploy daemon haproxy.nfs.foo.smithi193.ezysox ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.338422Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.qsjxxv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-qsjxxv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.qsjxxv\nDeploy daemon haproxy.nfs.foo.smithi097.qsjxxv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:46:01.342983Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.dbhnqi on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:47:04.888 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:47:05.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:05 smithi193 bash[28559]: cluster 2023-11-17T07:47:03.669653+0000 mgr.smithi097.swvonl (mgr.14180) 2620 : cluster [DBG] pgmap v1843: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:05.423 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:05 smithi193 bash[28559]: audit 2023-11-17T07:47:03.916876+0000 mgr.smithi097.swvonl (mgr.14180) 2621 : audit [DBG] from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:05.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:05 smithi097 bash[24958]: cluster 2023-11-17T07:47:03.669653+0000 mgr.smithi097.swvonl (mgr.14180) 2620 : cluster [DBG] pgmap v1843: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:05.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:05 smithi097 bash[24958]: audit 2023-11-17T07:47:03.916876+0000 mgr.smithi097.swvonl (mgr.14180) 2621 : audit [DBG] from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:05.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:47:06.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:06 smithi097 bash[24958]: audit 2023-11-17T07:47:05.503310+0000 mon.smithi097 (mon.0) 1341 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rjtocq"}]: dispatch 2023-11-17T07:47:06.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:06 smithi193 bash[28559]: audit 2023-11-17T07:47:05.503310+0000 mon.smithi097 (mon.0) 1341 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.rjtocq"}]: dispatch 2023-11-17T07:47:07.570 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: cephadm 2023-11-17T07:47:05.502166+0000 mgr.smithi097.swvonl (mgr.14180) 2622 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.570 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.570 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.570 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.570 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.571 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.572 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: cephadm 2023-11-17T07:47:05.502694+0000 mgr.smithi097.swvonl (mgr.14180) 2623 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: cephadm 2023-11-17T07:47:05.504973+0000 mgr.smithi097.swvonl (mgr.14180) 2624 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.573 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: cephadm 2023-11-17T07:47:05.508704+0000 mgr.smithi097.swvonl (mgr.14180) 2625 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.edailh on smithi097 2023-11-17T07:47:07.574 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:07 smithi097 bash[24958]: cluster 2023-11-17T07:47:05.670871+0000 mgr.smithi097.swvonl (mgr.14180) 2626 : cluster [DBG] pgmap v1844: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: cephadm 2023-11-17T07:47:05.502166+0000 mgr.smithi097.swvonl (mgr.14180) 2622 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:47:07.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: cephadm 2023-11-17T07:47:05.502694+0000 mgr.smithi097.swvonl (mgr.14180) 2623 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: cephadm 2023-11-17T07:47:05.504973+0000 mgr.smithi097.swvonl (mgr.14180) 2624 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.rjtocq ... 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:07.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:07.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:07.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: cephadm 2023-11-17T07:47:05.508704+0000 mgr.smithi097.swvonl (mgr.14180) 2625 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.edailh on smithi097 2023-11-17T07:47:07.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:07 smithi193 bash[28559]: cluster 2023-11-17T07:47:05.670871+0000 mgr.smithi097.swvonl (mgr.14180) 2626 : cluster [DBG] pgmap v1844: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:09.575 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:09 smithi097 bash[24958]: cluster 2023-11-17T07:47:07.671705+0000 mgr.smithi097.swvonl (mgr.14180) 2627 : cluster [DBG] pgmap v1845: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:09.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:09 smithi193 bash[28559]: cluster 2023-11-17T07:47:07.671705+0000 mgr.smithi097.swvonl (mgr.14180) 2627 : cluster [DBG] pgmap v1845: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 198 B/s rd, 0 op/s 2023-11-17T07:47:10.012 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:47:10.012 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:05.504773Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\nDeploy daemon haproxy.nfs.foo.smithi193.rjtocq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.469256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\nDeploy daemon haproxy.nfs.foo.smithi097.edailh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.471975Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:47:10.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cephadm 2023-11-17T07:47:09.467029+0000 mgr.smithi097.swvonl (mgr.14180) 2628 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:47:10.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cephadm 2023-11-17T07:47:09.467541+0000 mgr.smithi097.swvonl (mgr.14180) 2629 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: audit 2023-11-17T07:47:09.468187+0000 mon.smithi097 (mon.0) 1342 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.edailh"}]: dispatch 2023-11-17T07:47:10.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cephadm 2023-11-17T07:47:09.469416+0000 mgr.smithi097.swvonl (mgr.14180) 2630 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cephadm 2023-11-17T07:47:09.472102+0000 mgr.smithi097.swvonl (mgr.14180) 2631 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cephadm 2023-11-17T07:47:09.474282+0000 mgr.smithi097.swvonl (mgr.14180) 2632 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: cluster 2023-11-17T07:47:09.475691+0000 mgr.smithi097.swvonl (mgr.14180) 2633 : cluster [DBG] pgmap v1846: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:47:10.805 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:10 smithi097 bash[24958]: audit 2023-11-17T07:47:10.001039+0000 mgr.smithi097.swvonl (mgr.14180) 2634 : audit [DBG] from='client.15760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cephadm 2023-11-17T07:47:09.467029+0000 mgr.smithi097.swvonl (mgr.14180) 2628 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.807 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:47:10.808 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.809 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cephadm 2023-11-17T07:47:09.467541+0000 mgr.smithi097.swvonl (mgr.14180) 2629 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: audit 2023-11-17T07:47:09.468187+0000 mon.smithi097 (mon.0) 1342 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.edailh"}]: dispatch 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cephadm 2023-11-17T07:47:09.469416+0000 mgr.smithi097.swvonl (mgr.14180) 2630 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.810 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.edailh ... 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.811 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cephadm 2023-11-17T07:47:09.472102+0000 mgr.smithi097.swvonl (mgr.14180) 2631 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cephadm 2023-11-17T07:47:09.474282+0000 mgr.smithi097.swvonl (mgr.14180) 2632 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: cluster 2023-11-17T07:47:09.475691+0000 mgr.smithi097.swvonl (mgr.14180) 2633 : cluster [DBG] pgmap v1846: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:47:10.812 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:10 smithi193 bash[28559]: audit 2023-11-17T07:47:10.001039+0000 mgr.smithi097.swvonl (mgr.14180) 2634 : audit [DBG] from='client.15760 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:11.050 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:47:11.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:11 smithi097 bash[24958]: cluster 2023-11-17T07:47:10.312821+0000 mon.smithi097 (mon.0) 1343 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:47:11.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:11 smithi097 bash[24958]: audit 2023-11-17T07:47:10.574387+0000 mon.smithi097 (mon.0) 1344 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:11.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:11 smithi193 bash[28559]: cluster 2023-11-17T07:47:10.312821+0000 mon.smithi097 (mon.0) 1343 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:47:11.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:11 smithi193 bash[28559]: audit 2023-11-17T07:47:10.574387+0000 mon.smithi097 (mon.0) 1344 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:47:12.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:47:13.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:13 smithi097 bash[24958]: cluster 2023-11-17T07:47:11.477028+0000 mgr.smithi097.swvonl (mgr.14180) 2635 : cluster [DBG] pgmap v1847: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:47:13.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:13 smithi193 bash[28559]: cluster 2023-11-17T07:47:11.477028+0000 mgr.smithi097.swvonl (mgr.14180) 2635 : cluster [DBG] pgmap v1847: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2023-11-17T07:47:15.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:15 smithi097 bash[24958]: cluster 2023-11-17T07:47:13.478134+0000 mgr.smithi097.swvonl (mgr.14180) 2636 : cluster [DBG] pgmap v1848: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:15.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:15 smithi193 bash[28559]: cluster 2023-11-17T07:47:13.478134+0000 mgr.smithi097.swvonl (mgr.14180) 2636 : cluster [DBG] pgmap v1848: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:15.791 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:47:15.791 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:05.504773Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\nDeploy daemon haproxy.nfs.foo.smithi193.rjtocq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.469256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\nDeploy daemon haproxy.nfs.foo.smithi097.edailh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.471975Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:47:16.701 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:47:17.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:17 smithi097 bash[24958]: cluster 2023-11-17T07:47:15.479583+0000 mgr.smithi097.swvonl (mgr.14180) 2637 : cluster [DBG] pgmap v1849: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:17.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:17 smithi097 bash[24958]: audit 2023-11-17T07:47:15.779635+0000 mgr.smithi097.swvonl (mgr.14180) 2638 : audit [DBG] from='client.15764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:17.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:17 smithi193 bash[28559]: cluster 2023-11-17T07:47:15.479583+0000 mgr.smithi097.swvonl (mgr.14180) 2637 : cluster [DBG] pgmap v1849: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:17.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:17 smithi193 bash[28559]: audit 2023-11-17T07:47:15.779635+0000 mgr.smithi097.swvonl (mgr.14180) 2638 : audit [DBG] from='client.15764 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:17.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:47:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:19 smithi097 bash[24958]: cluster 2023-11-17T07:47:17.480589+0000 mgr.smithi097.swvonl (mgr.14180) 2639 : cluster [DBG] pgmap v1850: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:19.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:19 smithi193 bash[28559]: cluster 2023-11-17T07:47:17.480589+0000 mgr.smithi097.swvonl (mgr.14180) 2639 : cluster [DBG] pgmap v1850: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:21.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:21 smithi097 bash[24958]: cluster 2023-11-17T07:47:19.481678+0000 mgr.smithi097.swvonl (mgr.14180) 2640 : cluster [DBG] pgmap v1851: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:21.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:21 smithi193 bash[28559]: cluster 2023-11-17T07:47:19.481678+0000 mgr.smithi097.swvonl (mgr.14180) 2640 : cluster [DBG] pgmap v1851: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 173 B/s rd, 0 op/s 2023-11-17T07:47:21.667 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:47:21.667 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:05.504773Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\nDeploy daemon haproxy.nfs.foo.smithi193.rjtocq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.469256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\nDeploy daemon haproxy.nfs.foo.smithi097.edailh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.471975Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:47:22.595 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:47:23.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:23 smithi097 bash[24958]: cluster 2023-11-17T07:47:21.482838+0000 mgr.smithi097.swvonl (mgr.14180) 2641 : cluster [DBG] pgmap v1852: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:47:23.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:23 smithi097 bash[24958]: audit 2023-11-17T07:47:21.660636+0000 mgr.smithi097.swvonl (mgr.14180) 2642 : audit [DBG] from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:23.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:23 smithi193 bash[28559]: cluster 2023-11-17T07:47:21.482838+0000 mgr.smithi097.swvonl (mgr.14180) 2641 : cluster [DBG] pgmap v1852: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:47:23.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:23 smithi193 bash[28559]: audit 2023-11-17T07:47:21.660636+0000 mgr.smithi097.swvonl (mgr.14180) 2642 : audit [DBG] from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:47:23.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:47:25.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:25 smithi097 bash[24958]: cluster 2023-11-17T07:47:23.483839+0000 mgr.smithi097.swvonl (mgr.14180) 2643 : cluster [DBG] pgmap v1853: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:25.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:25 smithi193 bash[28559]: cluster 2023-11-17T07:47:23.483839+0000 mgr.smithi097.swvonl (mgr.14180) 2643 : cluster [DBG] pgmap v1853: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:27.388 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:47:27.388 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:05.504773Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\nDeploy daemon haproxy.nfs.foo.smithi193.rjtocq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.469256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\nDeploy daemon haproxy.nfs.foo.smithi097.edailh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.471975Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:47:27.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:27 smithi097 bash[24958]: cluster 2023-11-17T07:47:25.484824+0000 mgr.smithi097.swvonl (mgr.14180) 2644 : cluster [DBG] pgmap v1854: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:27.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:27 smithi193 bash[28559]: cluster 2023-11-17T07:47:25.484824+0000 mgr.smithi097.swvonl (mgr.14180) 2644 : cluster [DBG] pgmap v1854: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:47:28.368 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:47:29.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:29 smithi193 bash[28559]: audit 2023-11-17T07:47:27.374765+0000 mgr.smithi097.swvonl (mgr.14180) 2645 : audit [DBG] from='client.15772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:29 smithi193 bash[28559]: cluster 2023-11-17T07:47:27.485700+0000 mgr.smithi097.swvonl (mgr.14180) 2646 : cluster [DBG] pgmap v1855: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:31 smithi193 bash[28559]: cluster 2023-11-17T07:47:29.486819+0000 mgr.smithi097.swvonl (mgr.14180) 2647 : cluster [DBG] pgmap v1856: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:33 smithi193 bash[28559]: cluster 2023-11-17T07:47:31.488122+0000 mgr.smithi097.swvonl (mgr.14180) 2648 : cluster [DBG] pgmap v1857: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:35 smithi193 bash[28559]: cluster 2023-11-17T07:47:33.489147+0000 mgr.smithi097.swvonl (mgr.14180) 2649 : cluster [DBG] pgmap v1858: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:37 smithi193 bash[28559]: cluster 2023-11-17T07:47:35.490560+0000 mgr.smithi097.swvonl (mgr.14180) 2650 : cluster [DBG] pgmap v1859: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:38 smithi193 bash[28559]: cluster 2023-11-17T07:47:37.491470+0000 mgr.smithi097.swvonl (mgr.14180) 2651 : cluster [DBG] pgmap v1860: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.613 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:40 smithi193 bash[28559]: cluster 2023-11-17T07:47:39.492564+0000 mgr.smithi097.swvonl (mgr.14180) 2652 : cluster [DBG] pgmap v1861: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:43 smithi193 bash[28559]: cluster 2023-11-17T07:47:41.493967+0000 mgr.smithi097.swvonl (mgr.14180) 2653 : cluster [DBG] pgmap v1862: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:45 smithi193 bash[28559]: cluster 2023-11-17T07:47:43.494670+0000 mgr.smithi097.swvonl (mgr.14180) 2654 : cluster [DBG] pgmap v1863: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:47 smithi193 bash[28559]: cluster 2023-11-17T07:47:45.495989+0000 mgr.smithi097.swvonl (mgr.14180) 2655 : cluster [DBG] pgmap v1864: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:49 smithi193 bash[28559]: cluster 2023-11-17T07:47:47.496880+0000 mgr.smithi097.swvonl (mgr.14180) 2656 : cluster [DBG] pgmap v1865: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:51 smithi193 bash[28559]: cluster 2023-11-17T07:47:49.497995+0000 mgr.smithi097.swvonl (mgr.14180) 2657 : cluster [DBG] pgmap v1866: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:53 smithi193 bash[28559]: cluster 2023-11-17T07:47:51.499378+0000 mgr.smithi097.swvonl (mgr.14180) 2658 : cluster [DBG] pgmap v1867: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:55 smithi193 bash[28559]: cluster 2023-11-17T07:47:53.500445+0000 mgr.smithi097.swvonl (mgr.14180) 2659 : cluster [DBG] pgmap v1868: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:57 smithi193 bash[28559]: cluster 2023-11-17T07:47:55.501869+0000 mgr.smithi097.swvonl (mgr.14180) 2660 : cluster [DBG] pgmap v1869: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:47:59 smithi193 bash[28559]: cluster 2023-11-17T07:47:57.502860+0000 mgr.smithi097.swvonl (mgr.14180) 2661 : cluster [DBG] pgmap v1870: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.614 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:01 smithi193 bash[28559]: cluster 2023-11-17T07:47:59.503958+0000 mgr.smithi097.swvonl (mgr.14180) 2662 : cluster [DBG] pgmap v1871: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:03 smithi193 bash[28559]: cluster 2023-11-17T07:48:01.505294+0000 mgr.smithi097.swvonl (mgr.14180) 2663 : cluster [DBG] pgmap v1872: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:05 smithi193 bash[28559]: cluster 2023-11-17T07:48:03.506398+0000 mgr.smithi097.swvonl (mgr.14180) 2664 : cluster [DBG] pgmap v1873: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:06 smithi193 bash[28559]: cluster 2023-11-17T07:48:05.507824+0000 mgr.smithi097.swvonl (mgr.14180) 2665 : cluster [DBG] pgmap v1874: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:29 smithi097 bash[24958]: audit 2023-11-17T07:47:27.374765+0000 mgr.smithi097.swvonl (mgr.14180) 2645 : audit [DBG] from='client.15772 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:29 smithi097 bash[24958]: cluster 2023-11-17T07:47:27.485700+0000 mgr.smithi097.swvonl (mgr.14180) 2646 : cluster [DBG] pgmap v1855: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:31 smithi097 bash[24958]: cluster 2023-11-17T07:47:29.486819+0000 mgr.smithi097.swvonl (mgr.14180) 2647 : cluster [DBG] pgmap v1856: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:33 smithi097 bash[24958]: cluster 2023-11-17T07:47:31.488122+0000 mgr.smithi097.swvonl (mgr.14180) 2648 : cluster [DBG] pgmap v1857: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:35 smithi097 bash[24958]: cluster 2023-11-17T07:47:33.489147+0000 mgr.smithi097.swvonl (mgr.14180) 2649 : cluster [DBG] pgmap v1858: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:37 smithi097 bash[24958]: cluster 2023-11-17T07:47:35.490560+0000 mgr.smithi097.swvonl (mgr.14180) 2650 : cluster [DBG] pgmap v1859: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:38 smithi097 bash[24958]: cluster 2023-11-17T07:47:37.491470+0000 mgr.smithi097.swvonl (mgr.14180) 2651 : cluster [DBG] pgmap v1860: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:40 smithi097 bash[24958]: cluster 2023-11-17T07:47:39.492564+0000 mgr.smithi097.swvonl (mgr.14180) 2652 : cluster [DBG] pgmap v1861: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.615 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:43 smithi097 bash[24958]: cluster 2023-11-17T07:47:41.493967+0000 mgr.smithi097.swvonl (mgr.14180) 2653 : cluster [DBG] pgmap v1862: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:45 smithi097 bash[24958]: cluster 2023-11-17T07:47:43.494670+0000 mgr.smithi097.swvonl (mgr.14180) 2654 : cluster [DBG] pgmap v1863: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:47 smithi097 bash[24958]: cluster 2023-11-17T07:47:45.495989+0000 mgr.smithi097.swvonl (mgr.14180) 2655 : cluster [DBG] pgmap v1864: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:49 smithi097 bash[24958]: cluster 2023-11-17T07:47:47.496880+0000 mgr.smithi097.swvonl (mgr.14180) 2656 : cluster [DBG] pgmap v1865: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:51 smithi097 bash[24958]: cluster 2023-11-17T07:47:49.497995+0000 mgr.smithi097.swvonl (mgr.14180) 2657 : cluster [DBG] pgmap v1866: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:53 smithi097 bash[24958]: cluster 2023-11-17T07:47:51.499378+0000 mgr.smithi097.swvonl (mgr.14180) 2658 : cluster [DBG] pgmap v1867: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:55 smithi097 bash[24958]: cluster 2023-11-17T07:47:53.500445+0000 mgr.smithi097.swvonl (mgr.14180) 2659 : cluster [DBG] pgmap v1868: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:57 smithi097 bash[24958]: cluster 2023-11-17T07:47:55.501869+0000 mgr.smithi097.swvonl (mgr.14180) 2660 : cluster [DBG] pgmap v1869: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:47:59 smithi097 bash[24958]: cluster 2023-11-17T07:47:57.502860+0000 mgr.smithi097.swvonl (mgr.14180) 2661 : cluster [DBG] pgmap v1870: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:01 smithi097 bash[24958]: cluster 2023-11-17T07:47:59.503958+0000 mgr.smithi097.swvonl (mgr.14180) 2662 : cluster [DBG] pgmap v1871: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:03 smithi097 bash[24958]: cluster 2023-11-17T07:48:01.505294+0000 mgr.smithi097.swvonl (mgr.14180) 2663 : cluster [DBG] pgmap v1872: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:05 smithi097 bash[24958]: cluster 2023-11-17T07:48:03.506398+0000 mgr.smithi097.swvonl (mgr.14180) 2664 : cluster [DBG] pgmap v1873: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:08.616 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:06 smithi097 bash[24958]: cluster 2023-11-17T07:48:05.507824+0000 mgr.smithi097.swvonl (mgr.14180) 2665 : cluster [DBG] pgmap v1874: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:09.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:09 smithi097 bash[24958]: cluster 2023-11-17T07:48:07.508887+0000 mgr.smithi097.swvonl (mgr.14180) 2666 : cluster [DBG] pgmap v1875: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:09.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:09 smithi193 bash[28559]: cluster 2023-11-17T07:48:07.508887+0000 mgr.smithi097.swvonl (mgr.14180) 2666 : cluster [DBG] pgmap v1875: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.478530+0000 mon.smithi097 (mon.0) 1345 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.773567+0000 mon.smithi097 (mon.0) 1346 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.774619+0000 mon.smithi097 (mon.0) 1347 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.782010+0000 mon.smithi097 (mon.0) 1348 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.788172+0000 mon.smithi097 (mon.0) 1349 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:48:10.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:10 smithi097 bash[24958]: audit 2023-11-17T07:48:09.799830+0000 mon.smithi097 (mon.0) 1350 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.478530+0000 mon.smithi097 (mon.0) 1345 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T07:48:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.773567+0000 mon.smithi097 (mon.0) 1346 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T07:48:10.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.774619+0000 mon.smithi097 (mon.0) 1347 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-17T07:48:10.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.782010+0000 mon.smithi097 (mon.0) 1348 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:10.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.788172+0000 mon.smithi097 (mon.0) 1349 : audit [DBG] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-17T07:48:10.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:10 smithi193 bash[28559]: audit 2023-11-17T07:48:09.799830+0000 mon.smithi097 (mon.0) 1350 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:11 smithi097 bash[24958]: cluster 2023-11-17T07:48:09.509794+0000 mgr.smithi097.swvonl (mgr.14180) 2667 : cluster [DBG] pgmap v1876: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:11 smithi097 bash[24958]: cluster 2023-11-17T07:48:09.782884+0000 mgr.smithi097.swvonl (mgr.14180) 2668 : cluster [DBG] pgmap v1877: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:11 smithi097 bash[24958]: cephadm 2023-11-17T07:48:09.803488+0000 mgr.smithi097.swvonl (mgr.14180) 2669 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jovjjk on smithi193 2023-11-17T07:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:11 smithi097 bash[24958]: cluster 2023-11-17T07:48:10.268952+0000 mon.smithi097 (mon.0) 1351 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:48:11.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:11 smithi097 bash[24958]: cluster 2023-11-17T07:48:10.269037+0000 mon.smithi097 (mon.0) 1352 : cluster [INF] Cluster is now healthy 2023-11-17T07:48:11.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:11 smithi193 bash[28559]: cluster 2023-11-17T07:48:09.509794+0000 mgr.smithi097.swvonl (mgr.14180) 2667 : cluster [DBG] pgmap v1876: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-11-17T07:48:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:11 smithi193 bash[28559]: cluster 2023-11-17T07:48:09.782884+0000 mgr.smithi097.swvonl (mgr.14180) 2668 : cluster [DBG] pgmap v1877: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:11 smithi193 bash[28559]: cephadm 2023-11-17T07:48:09.803488+0000 mgr.smithi097.swvonl (mgr.14180) 2669 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi193.jovjjk on smithi193 2023-11-17T07:48:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:11 smithi193 bash[28559]: cluster 2023-11-17T07:48:10.268952+0000 mon.smithi097 (mon.0) 1351 : cluster [INF] Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 4 daemon(s)) 2023-11-17T07:48:11.556 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:11 smithi193 bash[28559]: cluster 2023-11-17T07:48:10.269037+0000 mon.smithi097 (mon.0) 1352 : cluster [INF] Cluster is now healthy 2023-11-17T07:48:12.459 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:48:12.459 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:46:01.344606Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.eezgbm on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:05.504773Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.rjtocq on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-rjtocq\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.rjtocq\nDeploy daemon haproxy.nfs.foo.smithi193.rjtocq ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.469256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.edailh on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-edailh\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.edailh\nDeploy daemon haproxy.nfs.foo.smithi097.edailh ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:47:09.471975Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.ubqzky on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:48:12.798 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:12 smithi097 bash[24958]: audit 2023-11-17T07:48:11.590883+0000 mon.smithi097 (mon.0) 1353 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:12.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:12 smithi097 bash[24958]: cluster 2023-11-17T07:48:11.783737+0000 mgr.smithi097.swvonl (mgr.14180) 2670 : cluster [DBG] pgmap v1878: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:12 smithi193 bash[28559]: audit 2023-11-17T07:48:11.590883+0000 mon.smithi097 (mon.0) 1353 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:13.055 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:12 smithi193 bash[28559]: cluster 2023-11-17T07:48:11.783737+0000 mgr.smithi097.swvonl (mgr.14180) 2670 : cluster [DBG] pgmap v1878: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:13.371 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:48:13.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:13 smithi097 bash[24958]: audit 2023-11-17T07:48:12.448262+0000 mgr.smithi097.swvonl (mgr.14180) 2671 : audit [DBG] from='client.15776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:13.894 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:13 smithi193 bash[28559]: audit 2023-11-17T07:48:12.448262+0000 mgr.smithi097.swvonl (mgr.14180) 2671 : audit [DBG] from='client.15776 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:14.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ed3c8e926b9de93026c8a7c1d86be5d6821b52ac shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 -- ceph orch ls -f json 2023-11-17T07:48:14.799 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: cluster 2023-11-17T07:48:13.784563+0000 mgr.smithi097.swvonl (mgr.14180) 2672 : cluster [DBG] pgmap v1879: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: cephadm 2023-11-17T07:48:13.970843+0000 mgr.smithi097.swvonl (mgr.14180) 2673 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.800 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:48:14.801 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:14.802 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: cephadm 2023-11-17T07:48:13.971330+0000 mgr.smithi097.swvonl (mgr.14180) 2674 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: audit 2023-11-17T07:48:13.971950+0000 mon.smithi097 (mon.0) 1354 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jovjjk"}]: dispatch 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: cephadm 2023-11-17T07:48:13.973472+0000 mgr.smithi097.swvonl (mgr.14180) 2675 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jovjjk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:14.803 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:14.804 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:14 smithi097 bash[24958]: cephadm 2023-11-17T07:48:13.977594+0000 mgr.smithi097.swvonl (mgr.14180) 2676 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.rxpiwv on smithi097 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: cluster 2023-11-17T07:48:13.784563+0000 mgr.smithi097.swvonl (mgr.14180) 2672 : cluster [DBG] pgmap v1879: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: cephadm 2023-11-17T07:48:13.970843+0000 mgr.smithi097.swvonl (mgr.14180) 2673 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.056 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:48:15.057 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.058 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.059 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: cephadm 2023-11-17T07:48:13.971330+0000 mgr.smithi097.swvonl (mgr.14180) 2674 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.060 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: audit 2023-11-17T07:48:13.971950+0000 mon.smithi097 (mon.0) 1354 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi193.jovjjk"}]: dispatch 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: cephadm 2023-11-17T07:48:13.973472+0000 mgr.smithi097.swvonl (mgr.14180) 2675 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi193.jovjjk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi193.jovjjk ... 2023-11-17T07:48:15.061 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr 2.3: Pulling from library/haproxy 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:15.062 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:14 smithi193 bash[28559]: cephadm 2023-11-17T07:48:13.977594+0000 mgr.smithi097.swvonl (mgr.14180) 2676 : cephadm [INF] Deploying daemon haproxy.nfs.foo.smithi097.rxpiwv on smithi097 2023-11-17T07:48:17.299 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:16 smithi097 bash[24958]: cluster 2023-11-17T07:48:15.785634+0000 mgr.smithi097.swvonl (mgr.14180) 2677 : cluster [DBG] pgmap v1880: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:17.305 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:16 smithi193 bash[28559]: cluster 2023-11-17T07:48:15.785634+0000 mgr.smithi097.swvonl (mgr.14180) 2677 : cluster [DBG] pgmap v1880: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:18.524 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-11-17T07:48:18.525 INFO:teuthology.orchestra.run.smithi097.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2023-11-17T06:47:26.088545Z", "last_refresh": "2023-11-17T07:40:11.031859Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2023-11-17T06:47:16.835620Z", "last_refresh": "2023-11-17T07:40:10.726677Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2023-11-17T06:47:21.044333Z", "last_refresh": "2023-11-17T07:40:11.032053Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2023-11-17T07:47:09.474135Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.zmtnyq on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:48:13.973256Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi193.jovjjk on smithi193: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi193-jovjjk\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi193.jovjjk\nDeploy daemon haproxy.nfs.foo.smithi193.jovjjk ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr 2.3: Pulling from library/haproxy\nstat: stderr docker: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi193 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:48:18.034015Z service:ingress.nfs.foo [ERROR] \"Failed while placing haproxy.nfs.foo.smithi097.rxpiwv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv\nNon-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv\n/usr/bin/docker: stdout \n/usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv\nDeploy daemon haproxy.nfs.foo.smithi097.rxpiwv ...\nNon-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\nstat: stderr Unable to find image 'haproxy:2.3' locally\nstat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.\nstat: stderr See 'docker run --help'.\nERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib\"", "2023-11-17T07:48:18.036110Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi097.jwnhyj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\"", "2023-11-17T07:48:18.037674Z service:ingress.nfs.foo [ERROR] \"Failed while placing keepalived.nfs.foo.smithi193.pvoxxk on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo\""], "placement": {"count": 2}, "service_id": "nfs.foo", "service_name": "ingress.nfs.foo", "service_type": "ingress", "spec": {"backend_service": "nfs.foo", "frontend_port": 2049, "monitor_port": 9002, "virtual_ip": "10.0.31.97/16"}, "status": {"created": "2023-11-17T06:57:29.604200Z", "ports": [2049, 9002], "running": 0, "size": 4, "virtual_ip": "10.0.31.97/16"}}, {"events": ["2023-11-17T06:57:24.876873Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2023-11-17T06:57:24.868767Z", "last_refresh": "2023-11-17T07:40:10.727919Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2023-11-17T06:47:14.757133Z", "last_refresh": "2023-11-17T07:40:10.726923Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:08.695152Z service:mon [INFO] \"service was created\""], "placement": {"count": 2, "hosts": ["smithi097:172.21.15.97=smithi097", "smithi193:172.21.15.193=smithi193"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2023-11-17T06:49:08.688000Z", "last_refresh": "2023-11-17T07:40:10.727076Z", "running": 2, "size": 2}}, {"events": ["2023-11-17T06:57:29.603566Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 12049}, "status": {"created": "2023-11-17T06:57:29.598547Z", "last_refresh": "2023-11-17T07:40:10.728058Z", "ports": [12049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2023-11-17T06:47:23.575291Z", "last_refresh": "2023-11-17T07:40:10.727221Z", "ports": [9100], "running": 2, "size": 2}}, {"events": ["2023-11-17T06:49:39.832032Z service:osd.all-available-devices [INFO] \"service was created\""], "placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"created": "2023-11-17T06:49:39.818666Z", "last_refresh": "2023-11-17T07:40:10.727362Z", "running": 8, "size": 8}}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2023-11-17T06:47:18.906413Z", "last_refresh": "2023-11-17T07:40:11.032243Z", "ports": [9095], "running": 1, "size": 1}}] 2023-11-17T07:48:18.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:18 smithi097 bash[24958]: audit 2023-11-17T07:48:18.032929+0000 mon.smithi097 (mon.0) 1355 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.rxpiwv"}]: dispatch 2023-11-17T07:48:18.555 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:18 smithi193 bash[28559]: audit 2023-11-17T07:48:18.032929+0000 mon.smithi097 (mon.0) 1355 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' cmd=[{"prefix": "auth rm", "entity": "client.ingress.nfs.foo.smithi097.rxpiwv"}]: dispatch 2023-11-17T07:48:19.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cluster 2023-11-17T07:48:17.786430+0000 mgr.smithi097.swvonl (mgr.14180) 2678 : cluster [DBG] pgmap v1881: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:19.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cephadm 2023-11-17T07:48:18.032206+0000 mgr.smithi097.swvonl (mgr.14180) 2679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:19.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:19.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:19.546 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Traceback (most recent call last): 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:48:19.547 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: yield (conn, connr) 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: code, '\n'.join(err))) 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:19.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:19.549 INFO:tasks.cephadm:ingress.nfs.foo has 0/4 2023-11-17T07:48:19.550 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1119, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-11-17T07:48:21.375 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=94bd000e12d2439bb2871503647756bc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1119, in wait_for_service while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (301) after waiting for 300 seconds 2023-11-17T07:48:21.377 DEBUG:teuthology.run_tasks:Unwinding manager vip 2023-11-17T07:48:21.418 INFO:tasks.vip:Removing 10.0.15.97 (and any VIPs) on smithi097.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T07:48:21.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr del 10.0.15.97/16 dev enp3s0f1 2023-11-17T07:48:21.419 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cephadm 2023-11-17T07:48:18.032438+0000 mgr.smithi097.swvonl (mgr.14180) 2680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cephadm 2023-11-17T07:48:18.034157+0000 mgr.smithi097.swvonl (mgr.14180) 2681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.rxpiwv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.420 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:21.421 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.421 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.421 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stdout 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:21.422 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:21.423 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.423 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cephadm 2023-11-17T07:48:18.036202+0000 mgr.smithi097.swvonl (mgr.14180) 2682 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.jwnhyj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:48:21.424 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cephadm 2023-11-17T07:48:18.037760+0000 mgr.smithi097.swvonl (mgr.14180) 2683 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.pvoxxk on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:48:21.424 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cluster 2023-11-17T07:48:18.038518+0000 mgr.smithi097.swvonl (mgr.14180) 2684 : cluster [DBG] pgmap v1882: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 240 B/s rd, 0 op/s 2023-11-17T07:48:21.424 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:19 smithi097 bash[24958]: cluster 2023-11-17T07:48:18.265740+0000 mon.smithi097 (mon.0) 1356 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:48:21.424 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:20 smithi097 bash[24958]: audit 2023-11-17T07:48:18.512769+0000 mgr.smithi097.swvonl (mgr.14180) 2685 : audit [DBG] from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:21.438 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cluster 2023-11-17T07:48:17.786430+0000 mgr.smithi097.swvonl (mgr.14180) 2678 : cluster [DBG] pgmap v1881: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 199 B/s rd, 0 op/s 2023-11-17T07:48:21.438 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cephadm 2023-11-17T07:48:18.032206+0000 mgr.smithi097.swvonl (mgr.14180) 2679 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.438 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.438 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.438 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:21.439 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:21.440 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:21.440 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.440 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Traceback (most recent call last): 2023-11-17T07:48:21.440 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: yield (conn, connr) 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: code, '\n'.join(err))) 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.441 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:21.442 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cephadm 2023-11-17T07:48:18.032438+0000 mgr.smithi097.swvonl (mgr.14180) 2680 : cephadm [INF] Removing key for client.ingress.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cephadm 2023-11-17T07:48:18.034157+0000 mgr.smithi097.swvonl (mgr.14180) 2681 : cephadm [ERR] Failed while placing haproxy.nfs.foo.smithi097.rxpiwv on smithi097: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy-nfs-foo-smithi097-rxpiwv 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stdout 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: /usr/bin/docker: stderr Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-haproxy.nfs.foo.smithi097.rxpiwv 2023-11-17T07:48:21.443 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Deploy daemon haproxy.nfs.foo.smithi097.rxpiwv ... 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr Unable to find image 'haproxy:2.3' locally 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit. 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: stat: stderr See 'docker run --help'. 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: ERROR: Failed to extract uid/gid for path /var/lib: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/library/haproxy:2.3 -e NODE_NAME=smithi097 -e CEPH_USE_RANDOM_NONCE=1 docker.io/library/haproxy:2.3 -c %u %g /var/lib 2023-11-17T07:48:21.444 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cephadm 2023-11-17T07:48:18.036202+0000 mgr.smithi097.swvonl (mgr.14180) 2682 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi097.jwnhyj on smithi097: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:48:21.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cephadm 2023-11-17T07:48:18.037760+0000 mgr.smithi097.swvonl (mgr.14180) 2683 : cephadm [ERR] Failed while placing keepalived.nfs.foo.smithi193.pvoxxk on smithi193: Failed to generate keepalived.conf: No daemons deployed for ingress.nfs.foo 2023-11-17T07:48:21.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cluster 2023-11-17T07:48:18.038518+0000 mgr.smithi097.swvonl (mgr.14180) 2684 : cluster [DBG] pgmap v1882: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 240 B/s rd, 0 op/s 2023-11-17T07:48:21.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:19 smithi193 bash[28559]: cluster 2023-11-17T07:48:18.265740+0000 mon.smithi097 (mon.0) 1356 : cluster [WRN] Health check failed: Failed to place 4 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2023-11-17T07:48:21.445 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:20 smithi193 bash[28559]: audit 2023-11-17T07:48:18.512769+0000 mgr.smithi097.swvonl (mgr.14180) 2685 : audit [DBG] from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T07:48:21.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo ip addr del 10.0.31.97/16 dev enp3s0f1 2023-11-17T07:48:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:RTNETLINK answers: Cannot assign requested address 2023-11-17T07:48:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-17T07:48:21.513 INFO:tasks.vip:Removing 10.0.15.193 (and any VIPs) on smithi193.front.sepia.ceph.com iface enp3s0f1... 2023-11-17T07:48:21.513 DEBUG:teuthology.orchestra.run.smithi193:> sudo ip addr del 10.0.15.193/16 dev enp3s0f1 2023-11-17T07:48:21.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo ip addr del 10.0.31.97/16 dev enp3s0f1 2023-11-17T07:48:21.548 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 bash[24958]: cluster 2023-11-17T07:48:20.039874+0000 mgr.smithi097.swvonl (mgr.14180) 2686 : cluster [DBG] pgmap v1883: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 399 B/s rd, 0 op/s 2023-11-17T07:48:21.549 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 bash[24958]: audit 2023-11-17T07:48:20.626609+0000 mon.smithi097 (mon.0) 1357 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:21 smithi193 bash[28559]: cluster 2023-11-17T07:48:20.039874+0000 mgr.smithi097.swvonl (mgr.14180) 2686 : cluster [DBG] pgmap v1883: 97 pgs: 97 active+clean; 2.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 399 B/s rd, 0 op/s 2023-11-17T07:48:21.558 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:21 smithi193 bash[28559]: audit 2023-11-17T07:48:20.626609+0000 mon.smithi097 (mon.0) 1357 : audit [INF] from='mgr.14180 172.21.15.97:0/2427377552' entity='mgr.smithi097.swvonl' 2023-11-17T07:48:21.572 INFO:teuthology.orchestra.run.smithi193.stderr:RTNETLINK answers: Cannot assign requested address 2023-11-17T07:48:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-11-17T07:48:21.573 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-17T07:48:21.587 INFO:tasks.cephadm:Teardown begin 2023-11-17T07:48:21.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T07:48:21.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T07:48:21.630 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-17T07:48:21.630 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T07:48:21.653 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T07:48:21.676 INFO:tasks.cephadm:Stopping all daemons... 2023-11-17T07:48:21.677 INFO:tasks.cephadm.mon.smithi097:Stopping mon.smithi097... 2023-11-17T07:48:21.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097 2023-11-17T07:48:21.897 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 systemd[1]: Stopping Ceph mon.smithi097 for e74de5f6-8514-11ee-8dbf-212e2dc638e7... 2023-11-17T07:48:21.897 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 bash[132282]: Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-mon.smithi097 2023-11-17T07:48:22.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 bash[24958]: debug 2023-11-17T07:48:21.892+0000 7f17f94d2700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi097 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T07:48:22.195 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:Nov 17 07:48:21 smithi097 bash[24958]: debug 2023-11-17T07:48:21.892+0000 7f17f94d2700 -1 mon.smithi097@0(leader) e2 *** Got Signal Terminated *** 2023-11-17T07:48:23.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi097.service' 2023-11-17T07:48:23.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T07:48:23.241 INFO:tasks.cephadm.mon.smithi097:Stopped mon.smithi097 2023-11-17T07:48:23.241 INFO:tasks.cephadm.mon.smithi193:Stopping mon.smithi193... 2023-11-17T07:48:23.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi193 2023-11-17T07:48:23.528 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:23 smithi193 systemd[1]: Stopping Ceph mon.smithi193 for e74de5f6-8514-11ee-8dbf-212e2dc638e7... 2023-11-17T07:48:23.529 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:23 smithi193 bash[55603]: Error response from daemon: No such container: ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7-mon.smithi193 2023-11-17T07:48:23.805 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:23 smithi193 bash[28559]: debug 2023-11-17T07:48:23.524+0000 7fdd72e0a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.smithi193 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T07:48:23.806 INFO:journalctl@ceph.mon.smithi193.smithi193.stdout:Nov 17 07:48:23 smithi193 bash[28559]: debug 2023-11-17T07:48:23.524+0000 7fdd72e0a700 -1 mon.smithi193@1(peon) e2 *** Got Signal Terminated *** 2023-11-17T07:48:24.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e74de5f6-8514-11ee-8dbf-212e2dc638e7@mon.smithi193.service' 2023-11-17T07:48:24.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T07:48:24.564 INFO:tasks.cephadm.mon.smithi193:Stopped mon.smithi193 2023-11-17T07:48:24.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 --force --keep-logs 2023-11-17T07:49:28.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 --force --keep-logs 2023-11-17T07:50:27.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T07:50:27.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-17T07:50:27.462 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-17T07:50:27.465 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/ubuntu@smithi097.front.sepia.ceph.com/crash 2023-11-17T07:50:27.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash -- . 2023-11-17T07:50:27.506 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T07:50:27.506 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2023-11-17T07:50:27.507 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/ubuntu@smithi193.front.sepia.ceph.com/crash 2023-11-17T07:50:27.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash -- . 2023-11-17T07:50:27.522 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T07:50:27.522 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2023-11-17T07:50:27.523 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-17T07:50:27.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-11-17T07:50:27.567 INFO:teuthology.orchestra.run.smithi097.stderr:grep: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph.log: No such file or directory 2023-11-17T07:50:27.569 INFO:tasks.cephadm:Compressing logs... 2023-11-17T07:50:27.569 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T07:50:27.613 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T07:50:27.626 INFO:teuthology.orchestra.run.smithi097.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-11-17T07:50:27.629 INFO:teuthology.orchestra.run.smithi193.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-11-17T07:50:27.633 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T07:50:27.633 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi097.log 2023-11-17T07:50:27.634 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.5.log 2023-11-17T07:50:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.1.log 2023-11-17T07:50:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi097.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi097.swvonl.log 2023-11-17T07:50:27.635 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.5.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.3.log 2023-11-17T07:50:27.636 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log 2023-11-17T07:50:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi097.swvonl.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.7.log 2023-11-17T07:50:27.637 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.3.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi097.vbcetq.log 2023-11-17T07:50:27.646 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.7.log: 92.4% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log.gz 2023-11-17T07:50:27.646 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi097.vbcetq.log: 60.2% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi097.vbcetq.log.gz 2023-11-17T07:50:27.660 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T07:50:27.668 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T07:50:27.668 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.6.log 2023-11-17T07:50:27.669 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi193.uynvbw.log 2023-11-17T07:50:27.669 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi193.log 2023-11-17T07:50:27.670 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.6.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi193.uynvbw.log: 83.5% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mds.foofs.smithi193.uynvbw.log.gz 2023-11-17T07:50:27.670 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi193.zqrhit.log 2023-11-17T07:50:27.671 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.0.log 2023-11-17T07:50:27.671 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi193.zqrhit.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi193.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.4.log 2023-11-17T07:50:27.672 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log 2023-11-17T07:50:27.673 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.2.log 2023-11-17T07:50:27.681 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.4.log: /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.2.log: 92.3% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-volume.log.gz 2023-11-17T07:50:27.684 INFO:teuthology.orchestra.run.smithi193.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T07:50:27.684 INFO:teuthology.orchestra.run.smithi193.stderr: 94.1% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi193.zqrhit.log.gz 2023-11-17T07:50:28.313 INFO:teuthology.orchestra.run.smithi097.stderr: 90.0% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mgr.smithi097.swvonl.log.gz 2023-11-17T07:50:28.402 INFO:teuthology.orchestra.run.smithi193.stderr: 93.2% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi193.log.gz 2023-11-17T07:50:28.539 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.5.log.gz 2023-11-17T07:50:28.796 INFO:teuthology.orchestra.run.smithi193.stderr: 93.1% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.4.log.gz 2023-11-17T07:50:28.809 INFO:teuthology.orchestra.run.smithi097.stderr: 93.4% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.3.log.gz 2023-11-17T07:50:28.818 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.0.log.gz 2023-11-17T07:50:28.846 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.1.log.gz 2023-11-17T07:50:28.925 INFO:teuthology.orchestra.run.smithi097.stderr: 93.2% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.7.log.gz 2023-11-17T07:50:28.950 INFO:teuthology.orchestra.run.smithi193.stderr: 93.1% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.2.log.gz 2023-11-17T07:50:29.235 INFO:teuthology.orchestra.run.smithi193.stderr: 92.9% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-osd.6.log.gz 2023-11-17T07:50:29.236 INFO:teuthology.orchestra.run.smithi193.stderr: 2023-11-17T07:50:29.236 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m1.618s 2023-11-17T07:50:29.236 INFO:teuthology.orchestra.run.smithi193.stderr:user 0m5.729s 2023-11-17T07:50:29.237 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m0.181s 2023-11-17T07:50:29.302 INFO:teuthology.orchestra.run.smithi097.stderr: 89.9% -- replaced with /var/log/ceph/e74de5f6-8514-11ee-8dbf-212e2dc638e7/ceph-mon.smithi097.log.gz 2023-11-17T07:50:29.303 INFO:teuthology.orchestra.run.smithi097.stderr: 2023-11-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m1.688s 2023-11-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m6.709s 2023-11-17T07:50:29.304 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.273s 2023-11-17T07:50:29.305 INFO:tasks.cephadm:Archiving logs... 2023-11-17T07:50:29.305 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/ubuntu@smithi097.front.sepia.ceph.com/log 2023-11-17T07:50:29.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T07:50:30.289 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/ubuntu@smithi193.front.sepia.ceph.com/log 2023-11-17T07:50:30.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T07:50:30.994 INFO:tasks.cephadm:Removing cluster... 2023-11-17T07:50:30.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 --force 2023-11-17T07:50:32.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e74de5f6-8514-11ee-8dbf-212e2dc638e7 --force 2023-11-17T07:50:33.735 INFO:tasks.cephadm:Removing cephadm ... 2023-11-17T07:50:33.735 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T07:50:33.743 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T07:50:33.750 INFO:tasks.cephadm:Teardown complete 2023-11-17T07:50:33.750 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2023-11-17T07:50:33.780 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi097:/dev/vg_nvme/lv_1... 2023-11-17T07:50:33.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo nvme disconnect -n lv_1 2023-11-17T07:50:34.084 INFO:teuthology.orchestra.run.smithi097.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-11-17T07:50:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:34.085 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi097:/dev/vg_nvme/lv_2... 2023-11-17T07:50:34.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo nvme disconnect -n lv_2 2023-11-17T07:50:34.363 INFO:teuthology.orchestra.run.smithi097.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-11-17T07:50:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:34.365 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi097:/dev/vg_nvme/lv_3... 2023-11-17T07:50:34.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo nvme disconnect -n lv_3 2023-11-17T07:50:34.651 INFO:teuthology.orchestra.run.smithi097.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-11-17T07:50:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:34.653 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi097:/dev/vg_nvme/lv_4... 2023-11-17T07:50:34.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo nvme disconnect -n lv_4 2023-11-17T07:50:34.979 INFO:teuthology.orchestra.run.smithi097.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-11-17T07:50:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:34.981 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-11-17T07:50:34.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/scratch_devs 2023-11-17T07:50:34.996 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi193:/dev/vg_nvme/lv_1... 2023-11-17T07:50:34.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo nvme disconnect -n lv_1 2023-11-17T07:50:35.291 INFO:teuthology.orchestra.run.smithi193.stdout:NQN:lv_1 disconnected 1 controller(s) 2023-11-17T07:50:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:35.293 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi193:/dev/vg_nvme/lv_2... 2023-11-17T07:50:35.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo nvme disconnect -n lv_2 2023-11-17T07:50:35.579 INFO:teuthology.orchestra.run.smithi193.stdout:NQN:lv_2 disconnected 1 controller(s) 2023-11-17T07:50:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:35.581 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi193:/dev/vg_nvme/lv_3... 2023-11-17T07:50:35.581 DEBUG:teuthology.orchestra.run.smithi193:> sudo nvme disconnect -n lv_3 2023-11-17T07:50:35.851 INFO:teuthology.orchestra.run.smithi193.stdout:NQN:lv_3 disconnected 1 controller(s) 2023-11-17T07:50:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:35.853 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi193:/dev/vg_nvme/lv_4... 2023-11-17T07:50:35.853 DEBUG:teuthology.orchestra.run.smithi193:> sudo nvme disconnect -n lv_4 2023-11-17T07:50:36.143 INFO:teuthology.orchestra.run.smithi193.stdout:NQN:lv_4 disconnected 1 controller(s) 2023-11-17T07:50:36.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:36.145 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-11-17T07:50:36.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/scratch_devs 2023-11-17T07:50:36.160 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-17T07:50:36.169 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-17T07:50:36.170 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T07:50:36.172 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi097.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================== 2023-11-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi097.stdout:*hv01.front.sepi 44.4.53.2 2 u 50 128 377 0.099 -2.013 0.183 2023-11-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi097.stdout:+hv02.front.sepi 63.231.80.2 3 u 44 128 377 0.114 1.986 0.141 2023-11-17T07:50:36.192 INFO:teuthology.orchestra.run.smithi097.stdout:+hv03.front.sepi 74.6.168.72 3 u 31 128 377 0.116 1.972 0.134 2023-11-17T07:50:36.193 INFO:teuthology.orchestra.run.smithi097.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi193.stdout: remote refid st t when poll reach delay offset jitter 2023-11-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================== 2023-11-17T07:50:36.225 INFO:teuthology.orchestra.run.smithi193.stdout:*hv01.front.sepi 44.4.53.2 2 u 48 64 377 0.077 -2.133 0.161 2023-11-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi193.stdout:+hv02.front.sepi 63.231.80.2 3 u 47 64 377 0.075 1.623 0.083 2023-11-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi193.stdout:+hv03.front.sepi 74.6.168.72 3 u 44 64 377 0.069 1.707 0.088 2023-11-17T07:50:36.226 INFO:teuthology.orchestra.run.smithi193.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-11-17T07:50:36.227 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-17T07:50:36.238 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-17T07:50:36.239 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-17T07:50:36.249 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-17T07:50:36.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-17T07:50:36.269 INFO:teuthology.task.internal:Duration was 4297.002570 seconds 2023-11-17T07:50:36.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-17T07:50:36.280 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-17T07:50:36.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T07:50:36.283 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T07:50:36.330 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-17T07:50:36.330 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2023-11-17T07:50:36.330 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T07:50:36.385 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2023-11-17T07:50:36.385 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T07:50:36.399 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-17T07:50:36.399 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T07:50:36.433 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T07:50:36.530 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-17T07:50:36.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T07:50:36.533 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T07:50:36.773 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-17T07:50:36.784 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-17T07:50:36.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T07:50:36.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T07:50:36.815 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-17T07:50:36.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T07:50:36.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T07:50:36.857 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2023-11-17T07:50:36.872 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2023-11-17T07:50:36.896 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T07:50:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:36.932 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T07:50:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T07:50:36.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-17T07:50:36.955 INFO:teuthology.task.internal:Transferring archived files... 2023-11-17T07:50:36.955 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/smithi097 2023-11-17T07:50:36.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T07:50:37.013 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312/remote/smithi193 2023-11-17T07:50:37.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T07:50:37.041 INFO:teuthology.task.internal:Removing archive directory... 2023-11-17T07:50:37.041 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T07:50:37.049 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T07:50:37.085 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-17T07:50:37.097 INFO:teuthology.task.internal:Not uploading archives. 2023-11-17T07:50:37.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-17T07:50:37.108 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-17T07:50:37.109 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T07:50:37.111 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T07:50:37.116 INFO:teuthology.orchestra.run.smithi097.stdout: 1048602 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 07:50 /home/ubuntu/cephtest 2023-11-17T07:50:37.133 INFO:teuthology.orchestra.run.smithi193.stdout: 1048601 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 07:50 /home/ubuntu/cephtest 2023-11-17T07:50:37.136 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-17T07:50:37.146 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-17T07:50:37.168 INFO:teuthology.nuke:Checking targets against current locks 2023-11-17T07:50:37.195 DEBUG:teuthology.nuke:shortname: smithi097 2023-11-17T07:50:37.195 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T07:50:37.216 DEBUG:teuthology.nuke:shortname: smithi193 2023-11-17T07:50:37.217 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T07:50:37.220 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 06:29:53.640601', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-17T07:50:37.236 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 06:29:53.639356', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-11-17T07:50:37.265 INFO:teuthology.orchestra.console.smithi097:Power off 2023-11-17T07:50:37.265 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T07:50:37.289 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2023-11-17T07:50:37.290 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:37.305 INFO:teuthology.orchestra.console.smithi193:Power off 2023-11-17T07:50:37.306 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T07:50:37.315 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2023-11-17T07:50:37.328 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2023-11-17T07:50:37.328 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:37.352 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-11-17T07:50:41.316 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:41.353 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:41.427 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2023-11-17T07:50:41.462 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-11-17T07:50:45.430 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:45.464 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:45.541 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2023-11-17T07:50:45.575 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2023-11-17T07:50:49.546 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:49.577 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T07:50:49.656 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2023-11-17T07:50:49.687 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2023-11-17T07:50:49.758 INFO:teuthology.orchestra.console.smithi097:Power off completed 2023-11-17T07:50:49.788 INFO:teuthology.orchestra.console.smithi193:Power off completed 2023-11-17T07:50:49.892 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} duration: 4297.002570390701 failure_reason: reached maximum tries (301) after waiting for 300 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=94bd000e12d2439bb2871503647756bc status: fail success: false 2023-11-17T07:50:49.893 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T07:50:49.975 INFO:teuthology.run:FAIL